Hostname: page-component-669899f699-2mbcq Total loading time: 0 Render date: 2025-05-02T16:28:17.895Z Has data issue: false hasContentIssue false
  • English
  • Français

Is a non-representative convenience sample of adults good enough? Insights from an economic experiment

Published online by Cambridge University Press:  17 January 2025

Sean F. Ellis Open the ORCID record for Sean F. Ellis [Opens in a new window] ,
Olesya M. Savchenko  and
Kent D. Messer
Sean F. Ellis*
Affiliation:
Behavior Change for Good, The Wharton School, University of Pennsylvania, 3720 Walnut St., 19104 Philadelphia, PA, USA
Olesya M. Savchenko
Affiliation:
Food and Resource Economics Department, University of Florida, 1183 McCarty Hall A, 32611 Gainesville, FL, USA
Kent D. Messer*
Affiliation:
Department of Applied Economics and Statistics, University of Delaware, 531 S. College Avenue, 19716 Newark, DE, USA
*
e-mail: [email protected]
[email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Recruitment of representative and generalizable adult samples is a major challenge for researchers conducting economic field experiments. Limited access to representative samples or the high cost of obtaining them often leads to the recruitment of non-representative convenience samples. This research compares the findings from two field experiments involving 860 adults: one from a non-representative in-person convenience sample and one from a representative online counterpart. We find no meaningful differences in the key behaviors of interest between the two samples. These findings contribute to a growing body of literature demonstrating that non-representative convenience samples can be sufficient in certain contexts.

Keywords

Non-representative convenience samplingField experimentsOnline recruitmentRepresentative sampling

JEL classification

C93: Field Experiments B41: Economic Methodology C83: Survey Methods • Sampling Methods
Type
Methodology Paper
Information
Journal of the Economic Science Association , Volume 9 , Issue 2 , December 2023 , pp. 293 - 307
Copyright
Copyright © The Author(s), under exclusive licence to Economic Science Association 2023.

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Banerjee, A. V., Duflo, E. (2009). The experimental approach to development economics. Annual Review of Economics, 1(1), 151–178. 10.1146/annurev.economics.050708.143235CrossRefGoogle Scholar
Boas, T. C., Christenson, D. P., Glick, D. M. (2020). Recruiting large online samples in the United States and India: facebook, mechanical turk, and qualtrics. Political Science Research and Methods, 8(2), 232–250. 10.1017/psrm.2018.28CrossRefGoogle Scholar
Bol, D. (2019). Putting politics in the lab: a review of lab experiments in political science. Government and Opposition, 54(1), 167–190. 10.1017/gov.2018.14CrossRefGoogle Scholar
Camerer, C. (2011). The promise and success of lab-field generalizability in experimental economics: A critical reply to Levitt and List. SSRN Electronic Journal, 10.2139/ssrn.1977749CrossRefGoogle Scholar
Ellis, S. F., Savchenko, O. M., Messer, K. D. (2022). Mitigating stigma associated with recycled water: Aquifer recharge and trophic levels. American Journal of Agricultural Economics, 104(3), 1077–1099. 10.1111/ajae.12256CrossRefGoogle Scholar
Frigau, L., Medda, T., Pelligra, V. (2019). From the field to the lab. an experiment on the representativeness of standard laboratory subjects. Journal of Behavioral and Experimental Economics, 78, 160–169. 10.1016/j.socec.2018.06.003CrossRefGoogle Scholar
Gelman, A., Goal, S., Rothschild, D., Wang, W., & Schill, D. (2016). High-frequency polling with non-representative data Political communication in real time, New York: Routledge.Google Scholar
Goldberg, M., Van Der Linden, S., Ballew, M. T., Rosenthal, S. A., Leiserowitz, A. (2019). Convenient but biased? the reliability of convenience samples in research about attitudes toward climate change. Osfpreprints., 11, 4453.Google Scholar
Levitt, S. D., List, J. A. (2007). Viewpoint: on the generalizability of lab behaviour to the field. Canadian Journal of Economics/revue Canadienne D’économique, 40(2), 347–370. 10.1111/j.1365-2966.2007.00412.xCrossRefGoogle Scholar
Levitt, S. D., List, J. A. (2008). Homo economicus evolves. Science, 319(5865), 909–910. 10.1126/science.1153640CrossRefGoogle ScholarPubMed
List, J. A., Price, M. K. (2016). The use of field experiments in environmental and resource economics. Review of Environmental Economics and Policy, 10(2), 206–225. 10.1093/reep/rew008CrossRefGoogle Scholar
Muller, S. (2014). Randomised Trials for Policy: A Review of the External Validity of Treatment Effects (Working Paper). Accessed 14 Dec 2020.Google Scholar
Palm-Forster, L. H., Ferraro, P. J., Janusch, N., Vossler, C. A., Messer, K. D. (2019). Behavioral and experimental agri-environmental research: Methodological challenges, literature gaps, and recommendations. Environmental and Resource Economics, 73(3), 719–742. 10.1007/s10640-019-00342-xCrossRefGoogle Scholar
Palm-Forster, L. H., Messer, K. D., & Palm-Forster, L. H. (2021). Experimental and Behavioral Economics to Inform Agri-Environmental Programs and Policies Handbook of Agricultural Economics, USA: David Just and Chris Barrett.Google Scholar
Peth, D., Mußhoff, O. (2020). Comparing compliance behaviour of students and farmers an extra-laboratory experiment in the context of agri-environmental nudges in Germany. Journal of Agricultural Economics., 71(2), 601–615. 10.1111/1477-9552.12352CrossRefGoogle Scholar
Roe, B. E., Just, D. R. (2009). Internal and external validity in economics research: tradeoffs between experiments, field experiments, natural experiments, and field data. American Journal of Agricultural Economics, 91(5), 1266–1271. 10.1111/j.1467-8276.2009.01295.xCrossRefGoogle Scholar
Rosch, S., Raszap Skorbiansky, S., Weigel, C., Messer, K. D., Hellerstein, D. (2021). Barriers to using economic experiments in evidence-based agricultural policymaking. Applied Economic Perspectives and Policy, 43(2), 531–555. 10.1002/aepp.13091CrossRefGoogle Scholar
Savchenko, O., Kecinski, M., Li, T., Messer, K. D. (2018). reclaimed water and food production: cautionary tales from consumer research. Environmental Research., 170, 320–331. 10.1016/j.envres.2018.12.051CrossRefGoogle ScholarPubMed
Wang, W., Rothschild, D., Goel, S., Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31(3), 980–991. 10.1016/j.ijforecast.2014.06.001CrossRefGoogle Scholar
Weigel, C., Paul, L. A., Ferraro, P. J., Messer, K. D. (2021). Challenges in recruiting US farmers for policy-relevant economic field experiments. Applied Economic Perspectives and Policy, 43(2), 556–572. 10.1002/aepp.13066CrossRefGoogle Scholar