Hostname: page-component-669899f699-qzcqf Total loading time: 0 Render date: 2025-05-04T13:54:52.568Z Has data issue: false hasContentIssue false
  • English
  • Français

Temporally Dynamic, Cohort-Varying Value-Added Models

Published online by Cambridge University Press:  01 January 2025

Garritt L. Page ,
Ernesto San Martín Open the ORCID record for Ernesto San Martín [Opens in a new window] ,
David Torres Irribarra  and
Sébastien Van Bellegem
Garritt L. Page
Affiliation:
Brigham Young University
Ernesto San Martín*
Affiliation:
Millennium Nucleus on Intergenerational Mobility MOVI LIDAM/CORE, Université catholique de Louvain Pontificia Universidad Católica de Chile
David Torres Irribarra
Affiliation:
School of Psychology, Pontificia Universidad Católica de Chile
Sébastien Van Bellegem
Affiliation:
LIDAM/CORE, Université catholique de Louvain
*
Correspondence should be made to Ernesto SanMartín, Faculty of Mathematics, Pontificia Universidad Católica de Chile, Vicuna Mackenna 4860, Macul, Santiago, Chile. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

We aim to estimate school value-added dynamically in time. Our principal motivation for doing so is to establish school effectiveness persistence while taking into account the temporal dependence that typically exists in school performance from one year to the next. We propose two methods of incorporating temporal dependence in value-added models. In the first we model the random school effects that are commonly present in value-added models with an auto-regressive process. In the second approach, we incorporate dependence in value-added estimators by modeling the performance of one cohort based on the previous cohort’s performance. An identification analysis allows us to make explicit the meaning of the corresponding value-added indicators: based on these meanings, we show that each model is useful for monitoring specific aspects of school persistence. Furthermore, we carefully detail how value-added can be estimated over time. We show through simulations that ignoring temporal dependence when it exists results in diminished efficiency in value-added estimation while incorporating it results in improved estimation (even when temporal dependence is weak). Finally, we illustrate the methodology by considering two cohorts from Chile’s national standardized test in mathematics.

Keywords

School value persistenceValue-added modelsTemporal dependence
Type
Application Reviews and Case Studies
Information
Psychometrika , Volume 89 , Issue 3 , September 2024 , pp. 1074 - 1103
Copyright
Copyright © 2024 The Author(s), under exclusive licence to The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s11336-024-09979-0.

The manuscript was handled by the ARCS Editor Dr. Nidhi Kohl.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

References

Aitkin, M., Longford, N. (1986). Statistical modelling issues in school effectiveness studies. Journal of the Royal Statistical Society: Series A (General), 149(1), 1–26.CrossRefGoogle Scholar
Amrein-Beardsley, A., Holloway, J. (2019). Value-added models for teacher evaluation and accountability: Commonsense assumptions. Educational Policy, 33(3), 516–542.CrossRefGoogle Scholar
Ballou, D., Sanders, W., Wright, P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29(1), 37–65.CrossRefGoogle Scholar
Bellei, C., Vanni, X., Valenzuela, J. P., Contreras, D. (2016). School improvement trajectories: An empirical typology. School Effectiveness and School Improvement, 27, 275–292.CrossRefGoogle Scholar
Bianconcini, S., Cagnone, S. (2012). A general multivariate latent growth model with applications to student achievement. Journal of Educational and Behavioral Statistics, 37(2), 339–364.Google Scholar
Billingsley, P. (1968). Convergence of probability measures, Wiley.Google Scholar
Briggs, D. C., Weeks, J. P. (2011). The persistence of school-level value-added. Journal of Educational and Behavioral Statistics, 36, 616–637.CrossRefGoogle Scholar
Christensen, R., Johnson, W., Branscum, A., & Hanson, T. (2011). Bayesian ideas and data analysis: An introduction for scientists and statisticians. Taylor & Francis. https://books.google.com/books?id=qPERhCbePNcC.Google Scholar
Clarke, P., Crawford, C., Steele, F., Vignoles, A. (2015). Revisiting fixed-and random-effects models: Some considerations for policy-relevant education research. Education Economics, 23(3), 259–277.CrossRefGoogle Scholar
Ehlert, M., Koedel, C., Parsons, E., Podgursky, M. J. (2014). The sensitivity of value-added estimates to specification adjustments: Evidence from school- and teacher-level models in missouri. Statistics and Public Policy, 1(1), 19–27.CrossRefGoogle Scholar
Engle, R. E., Hendry, D. F., Richard, J. F. (1983). Exogeneity. Econometrica, 51, 277–304.CrossRefGoogle Scholar
EPI Briefing Paper. (2010). Problems with the use of student test scores to evaluate teachers. Economic Policy Institute.Google Scholar
Fariña, P., González, J., & San Martín, E. (2019). The use of an identifiability-based strategy for the interpretation of parameters in the 1PL-G and Rasch models. Psychometrika, 84, 511–528.CrossRefGoogle Scholar
Fisher, R. A. (1973). Statistical methods for research workers, Hafner Publishhing.Google Scholar
Fitzmaurice, G., Laird, N., Ware, J. (2004). Applied longitudinal analysis, Wiley.Google Scholar
Florens, J.-P., Marimoutou, V., Péguin-Feissolle, A. (2007). Econometric modeling and inference, Cambridge University Press.CrossRefGoogle Scholar
Gray, J., Goldstein, H., Jesson, D. (1996). Changes and improvements in schools’ effectiveness: Trends over five years. Research Papers in Education, 11, 35–51.CrossRefGoogle Scholar
Gray, J., Goldstein, H., Thomas, S. (2001). Predicting the future: The role of past performance in determining trends in institutional effectiveness at A level. British Educational Research Journal, 27, 391–405.CrossRefGoogle Scholar
Gray, J., Hopkins, D., Reynolds, D., Wilcox, B., Farrell, S., Jesson, D. (1999). Improving schools: Performance and potential, Open University Press.Google Scholar
Guldemond, H., Bosker, R. J. (2009). School effects on students’ progress: A dynamic perspective. School Effectiveness and School Improvement, 20(2), 255–268.CrossRefGoogle Scholar
Hanushek, E. A. (2020). Education production functions. In Bradley, S., Green, C. (Eds), The economics of education, Elsevier 161–170.CrossRefGoogle Scholar
Hsiao, C. (2014). Analysis of panel data, Cambridge University Press.CrossRefGoogle Scholar
Kinsler, J. (2012). Beyond levels and growth: Estimating teacher value-added and its persistence. Journal of Human Resources, 47(3), 722–753.CrossRefGoogle Scholar
Koedel, C., Mihaly, K., Rockoff, J. E. (2015). Value-added modeling: A review. Economics of Education Review, 47, 180–195.CrossRefGoogle Scholar
Kolmogorov, A. N. (1950). Foundations of the theory of probability, Chelsea Publishing Company.Google Scholar
Kyriakides, L., Georgiou, M. P., Creemers, B. P., Panayiotou, A., Reynolds, D. (2018). The impact of national educational policies on student achievement: A European study. School Effectiveness and School Improvement, 29(2), 171–203.CrossRefGoogle Scholar
Leckie, G. (2018). Avoiding bias when estimating the consistency and stability of value-added school effects. Journal of Educational and Behavioral Statistics, 43, 440–468.CrossRefGoogle Scholar
Lindley, D. V. (1983). Bayesian statistics: A review. In: CBMS-NSf regional conference series in applied mathematics, Philadelphia.Google Scholar
Liu, J., & Loeb, S. (2019). Engaging teachers: Measuring the impact of teachers on student attendance in secondary school. Journal of Human Resources, pp. 1216–8430R3.Google Scholar
Lockwood, J., McCaffrey, D. F., Mariano, L. T., Setodji, C. (2007). Bayesian methods for scalable multivariate value-added assessment. Journal of Educational and Behavioral Statistics, 32, 125–150.CrossRefGoogle Scholar
Longford, N. T. (2012). A revision of school effectiveness analysis. Journal of Educational and Behavioral Statistics, 37(1), 157–179.CrossRefGoogle Scholar
Lord, F. M., Novick, M. (1968). Statistical theories of mental test scores, Addison Wesley.Google Scholar
Manzi, J., & Preiss, D. (2013). Educational Assessment and Educational Achievement in South America. In J. Hattie & E. M. Anderman (Eds.), International guide to student achievement (p. chapter 9). Taylor and Friends.Google Scholar
Manzi, J., San Martín, E., Van Bellegem, S. (2014). School system evaluation by value added analysis under endogeneity. Psychometrika, 79(1), 130–153.CrossRefGoogle ScholarPubMed
McCaffey, D. F., Lockwood, J., Koretz, D. M., Louis, T. A., Hamilton, L. S. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67–101.CrossRefGoogle Scholar
Meckes, L., Carrasco, R. (2010). Two decades of simce: An overview of the national assessment system in Chile. Assessment in Education: Principles, Policy and Practice, 17, 233–248.Google Scholar
Mouchart, M., & Oulhaj, A. (2006). The role of exogenous randomness in the identification of conditional models. Metron - International Journal of Statistics, LXIV:253–271.Google Scholar
Neveu, J. (1972). Martingales ‘a temps discret. Paris: Masson et CIE.Google Scholar
Page, G. L. (2020). modernva: An implementation of two modern education-based value-added models [Computer software manual]. https://CRAN.R-project.org/package=modernVA (R-package version 0.1.1).Google Scholar
Page, G. L., San Martín, E., Orellana, J., González, J. (2017). Exploring complete school effectiveness via quantile value-added. Journal of the Royal Statistical Society Series A, 180, 315–340.CrossRefGoogle Scholar
Papay, J. P. (2011). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48(1), 163–193.CrossRefGoogle Scholar
Potthoff, R. F., Roy, S. N. (1964). A generalized multivariate analysis of variance model useful especially for growth curve problems. Biometrika, 51 3–4313–326.CrossRefGoogle Scholar
Reynolds, D., Sammons, P., Fraine, B. D., Damme, J. V., Townsend, T., Teddlie, C., Stringfield, S. (2014). Educational effectiveness research (EER): A state-of-the-art review. School Effectiveness and School Improvement, 25, 197–230.CrossRefGoogle Scholar
Rothstein, J. (2010). Teacher quality in educational production: Tracking, decay, and student achievement. The Quarterly Journal of Economics, 125(1), 175–214.CrossRefGoogle Scholar
Sanders, W. L., Horn, S. P. (1994). The Tennessee value-added assessment system (TVAAS): Mixed-model methodology in educational assessment. Journal of Personnel Evaluation in education, 8(3), 299–311.CrossRefGoogle Scholar
San Martín, E., González, J., Tuerlinckx, F. (2015). On the unidentifiability of the fixed-effects 3PL model. Psychometrika, 80, 450–467.CrossRefGoogle ScholarPubMed
San Martín, E., Jara, A., Rolin, J.-M., Mouchart, M. (2011). On the Bayesian nonparametric generalization of IRT-type models. Psychometrika, 76(3), 385–409.CrossRefGoogle Scholar
San Martín, E., Rolin, J.-M., Castro, L. M. (2013). Identification of the 1PL model with guessing parameter: Parametric and semi-parametric results. Psychometrika, 78, 341–379.Google ScholarPubMed
Sass, T. R., Hannaway, J., Xu, Z., Figlio, D. N., Feng, L. (2012). Value added of teachers in high-poverty schools and lower poverty schools. Journal of urban Economics, 72 2–3104–122.CrossRefGoogle Scholar
Scherrer, J. (2011). Measuring teaching using value-added modeling: The imperfect panacea. NASSP Bulletin, 95(2), 122–140.CrossRefGoogle Scholar
Strenio, J. F., Weisberg, H. I., & Bryk, A. S. (1983). Empirical bayes estimation of individual growth-curve parameters and their relationship to covariates. Biometrics, pp. 71–86.CrossRefGoogle Scholar
Tekwe, C. D., Carter, R. L., Ma, C.-X., Algina, J., Lucas, M. E., Roth, J., Resnick, M. B. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29(1), 11–36.CrossRefGoogle Scholar
Thomas, S., Peng, W. J., Gray, J. (2007). Modelling patterns of improvement over time: Value added trends in English secondary school performance across ten cohorts. Oxford Review of Education, 33, 261–295.CrossRefGoogle Scholar
Tymms, P., Merrell, C., Bailey, K. (2018). The long-term impact of effective teaching. School Effectiveness and School Improvement, 29(2), 242–261.CrossRefGoogle Scholar
Vanwynsberghe, G., Vanlaar, G., Van Damme, J., De Fraine, B. (2017). Long-term effects of primary schools on educational positions of students 2 and 4 years after the start of secondary education. School Effectiveness and School Improvement, 28(2), 167–190.CrossRefGoogle Scholar
Zimmerman, D. W. (1975). Probability spaces, hilbert spaces, and the axioms of test theory. Psychometrika, 40(3), 395–412.CrossRefGoogle Scholar
Supplementary material: File

Page et al. Supplementary material

Page et al. Supplementary material
Download Page et al. Supplementary material(File)
File 350.9 KB