Social scientists often use ranking questions to study people’s opinions and preferences. However, little is understood about the general nature of measurement errors in such questions, let alone their statistical consequences and what researchers can do about them. We introduce a statistical framework to improve ranking data analysis by addressing measurement errors in ranking questions. First, we characterize measurement errors from random responses—arbitrary and meaningless responses based on a wide range of random patterns. We then quantify bias due to random responses, show that the bias may change our conclusion in any direction, and clarify why item order randomization alone does not solve the statistical issue. Next, we introduce our methodology based on two key design-based considerations: item order randomization and the addition of an “anchor” ranking question with known correct answers. They allow researchers to (1) learn about the direction of the bias and (2) estimate the proportion of random responses, enabling our bias-corrected estimators. We illustrate our methods by studying the relative importance of people’s partisan identity compared to their racial, gender, and religious identities in American politics. We find that about 30% of respondents offered random responses and that these responses may affect our substantive conclusions.

Building on the availability of geospatial data, improvements in mapping software, and innovations in spatial statistics, political scientists are increasingly taking geography seriously. As we adopt the tools of geographers, we must also consider the methodological challenges they have identified. We focus on the modifiable areal unit problem (MAUP)—the idea that the size of aggregate spatial units and the location of their borders affect the empirical results we obtain. We first describe the logic of the MAUP, and then demonstrate the MAUP through simulations, showing MAUP-related inconsistency in regression results in randomly generated and real-world data. We identify MAUP concerns, and best practices, in top journals in political science. We conclude by suggesting how scholars may respond in theoretical and empirical terms to concerns about validity and reliability that arise from the MAUP.

Scholars often face a choice when designing conjoint experiments: to allow for or to exclude “odd” combinations of attribute levels in the randomized conjoint profiles shown to respondents (such as a profile of a Democratic candidate who does not support abortion rights or an individual who is a medical doctor but does not have a graduate degree). While previous work has studied the statistical and theoretical implications of this decision, there has been little effort to analyze how it impacts the behavior of survey respondents. Utilizing eye-tracking, this study considers how respondents’ attention, information search behavior, and choice patterns respond to odd combinations of attributes included in conjoint profiles. We find that the impact of odd attribute-level combinations is minimal. They do not impact attention, search, or choice behavior substantially or consistently. Our conclusion is that scholars should prioritize other considerations—such as statistical, theoretical, and substantive considerations—when designing conjoint experiments.

Public opinion surveys are vital for informing democratic decision-making, but responding to rapidly changing information environments and measuring beliefs within hard-to-reach communities can be challenging for traditional survey methods. This paper introduces a crowdsourced adaptive survey methodology (CSAS) that unites advances in natural language processing and adaptive algorithms to produce surveys that evolve with participant input. The CSAS method converts open-ended text provided by participants into survey items and applies a multi-armed bandit algorithm to determine which questions should be prioritized in the survey. The method’s adaptive nature allows new survey questions to be explored and imposes minimal costs in survey length. Applications in the domains of misinformation, issue salience, and local politics showcase CSAS’s ability to identify topics that might otherwise escape the notice of survey researchers. I conclude by highlighting CSAS’s potential to bridge conceptual gaps between researchers and participants in survey research.

We use instruction-tuned large language models (LLMs) like GPT-4, Llama 3, MiXtral, or Aya to position political texts within policy and ideological spaces. We ask an LLM where a tweet or a sentence of a political text stands on the focal dimension and take the average of the LLM responses to position political actors such as US Senators, or longer texts such as UK party manifestos or EU policy speeches given in 10 different languages. The correlations between the position estimates obtained with the best LLMs and benchmarks based on text coding by experts, crowdworkers, or roll call votes exceed .90. This approach is generally more accurate than the positions obtained with supervised classifiers trained on large amounts of research data. Using instruction-tuned LLMs to position texts in policy and ideological spaces is fast, cost-efficient, reliable, and reproducible (in the case of open LLMs) even if the texts are short and written in different languages. We conclude with cautionary notes about the need for empirical validation.

If instant runoff voting (IRV) mitigates strategic voting, is that because the rules of the system mechanically reduce strategic opportunities, or because of other more indirect effects? In single-vote plurality elections, a voter can be pivotal if adding one vote to a candidate would cause that candidate to win. In IRV, it is more complicated to identify when one voter can be pivotal. This letter derives all the ways that a single ballot can change the result of an IRV election, for any number of candidates and voters. I obtain an expression for the probability of casting a pivotal vote in IRV by phrasing the probability that any pivotal event occurs as a function of all the rankings cast by other voters. This expression facilitates modeling vote choice in IRV, and enables the estimation of voters’ strategic opportunities in IRV contests between any number of candidates. I present some illustrative simulations estimating pivotal probabilities in both IRV and Single-Member District Plurality for stylized electorates with identical preference structures. These simulations produce similar estimated pivotal probabilities in the two systems, suggesting that these systems may provide similar opportunities to strategically cast a decisive vote.

Past work on closed-ended survey responses demonstrates that inferring stable political attitudes requires separating signal from noise in “top of the head” answers to researchers’ questions. We outline a corresponding theory of the open-ended response, in which respondents make narrow, stand-in statements to convey more abstract, general attitudes. We then present a method designed to infer those attitudes. Our approach leverages co-variation with words used relatively frequently across respondents to infer what else they could have said without substantively changing what they meant—linking narrow themes to each other through associations with contextually prevalent words. This reflects the intuition that a respondent may use different specific statements at different points in time to convey similar meaning. We validate this approach using panel data in which respondents answer the same open-ended questions (concerning healthcare policy, most important problems, and evaluations of political parties) at multiple points in time, showing that our method’s output consistently exhibits higher within-subject correlations than hand-coding of narrow response categories, topic modeling, and large language model output. Finally, we show how large language models can be used to complement—but not, at present, substitute—our “implied word” method.

Large language models are a powerful tool for conducting text analysis in political science, but using them to annotate text has several drawbacks, including high cost, limited reproducibility, and poor explainability. Traditional supervised text classifiers are fast and reproducible, but require expensive hand annotation, which is especially difficult for rare classes. This article proposes using LLMs to generate synthetic training data for training smaller, traditional supervised text models. Synthetic data can augment limited hand annotated data or be used on its own to train a classifier with good performance and greatly reduced cost. I provide a conceptual overview of text generation, guidance on when researchers should prefer different techniques for generating synthetic text, a discussion of ethics, a simple technique for improving the quality of synthetic text, and an illustration of its limitations. I demonstrate the usefulness of synthetic training through three validations: synthetic news articles describing police responses to communal violence in India for training an event detection system, a multilingual corpus of synthetic populist manifesto statements for training a sentence-level populism classifier, and generating synthetic tweets describing the fighting in Ukraine to improve a named entity system.

Analysts often seek to compare representations in high-dimensional space, e.g., embedding vectors of the same word across groups. We show that the distance measures calculated in such cases can exhibit considerable statistical bias, that stems from uncertainty in the estimation of the elements of those vectors. This problem applies to Euclidean distance, cosine similarity, and other similar measures. After illustrating the severity of this problem for text-as-data applications, we provide and validate a bias correction for the squared Euclidean distance. This same correction also substantially reduces bias in ordinary Euclidean distance and cosine similarity estimates, but corrections for these measures are not quite unbiased and are (non-intuitively) bimodal when distances are close to zero. The estimators require obtaining the variance of the latent positions. We (will) implement the estimator in free software, and we offer recommendations for related work.

Regression discontinuity designs (RDD) are widely used in the social sciences to estimate causal effects from observational data. Following recent methodological advances, scholars can choose from various RDD estimators for point estimation and inference. This decision is mainly guided by theoretical results on optimality and Monte Carlo simulations because of a paucity of research on the performance of the different estimators in recovering real-world experimental benchmarks. Leveraging exact ties in personal votes in local elections in Colombia and Finland, which are resolved by a random lottery, we assess the performance of various estimators featuring different polynomial degrees, bias-correction methods, optimal bandwidths, and approaches to statistical inference. Using re-running and re-election as outcomes, we document only minor differences in the performance of the various implementation approaches when the conditional expectation function (CEF) of the outcomes in the vicinity of the discontinuity is close to linear. When approximating the curvature of the CEF is more challenging, bias-corrected and robust inference with coverage-error-rate-optimal bandwidths comes closer to the experimental benchmark than more widely used alternative implementations.

Confidence intervals are ubiquitous in the presentation of social science models, data, and effects. When several intervals are plotted together, one natural inclination is to ask whether the estimates represented by those intervals are significantly different from each other. Unfortunately, there is no general rule or procedure that would allow us to answer this question from the confidence intervals alone. It is well known that using the overlaps in 95% confidence intervals to perform significance tests at the 0.05 level does not work. Recent scholarship has developed and refined a set of tools for inferential confidence intervals that permit inference on confidence intervals with the appropriate type I error rate in many different bivariate contexts. These are all based on the same underlying idea of identifying the multiple of the standard error (i.e., a new confidence level) such that the overlap in confidence intervals matches the desired type I error rate. These procedures remain stymied by multiple simultaneous comparisons. We propose an entirely new procedure for developing inferential confidence intervals that decouples the testing and visualization that can overcome many of these problems in any visual testing scenario. We provide software in R and Stata to accomplish this goal.

Substantive research in the Social Sciences regularly investigates signed networks, where edges between actors are positive or negative. One often-studied example within International Relations for this type of network consists of countries that can cooperate with or fight against each other. These analyses often build on structural balance theory, one of the earliest and most prominent network theories. While the theorization and description of signed networks have made significant progress, the inferential study of link formation within them remains limited in the absence of appropriate statistical models. We fill this gap by proposing the Signed Exponential Random Graph Model (SERGM), extending the well-known Exponential Random Graph Model (ERGM) to networks where ties are not binary but positive or negative if a tie exists. Since most networks are dynamically evolving systems, we specify the model for both cross-sectional and dynamic networks. Based on hypotheses derived from structural balance theory, we formulate interpretable signed network statistics, capturing dynamics such as “the enemy of my enemy is my friend”. In our empirical application, we use the SERGM to analyze cooperation and conflict between countries within the international state system. We find evidence for structural balance in International Relations.

We develop a formal framework for accumulating evidence across studies and apply it to develop theoretical foundations for replication. Our primary contribution is to characterize the relationship between replication and distinct formulations of external validity. Whereas conventional wisdom holds that replication facilitates learning about external validity, we show that this is not, in general, the case. Our results show how comparisons of the magnitude or sign of empirical findings link to distinct concepts of external validity. However, without careful attention to the research design of constituent studies, replication can mislead efforts to assess external validity. We show that two studies must have essentially the same research designs, i.e., be harmonized, in order for their estimates to provide information about any kind of external validity. This result shows that even minor differences in research design between a study and its replication can introduce a discrepancy that is typically overlooked, a problem that becomes more pronounced as the number of studies increases. We conclude by outlining a design-driven approach to replication, which responds to the issues our framework identifies and details how a research agenda can manage them productively.