We examine the generalizability of single-topic studies, focusing on how often their confidence intervals capture the typical treatment effect from a larger population of possible studies. We show that the confidence intervals from these single-topic studies capture the typical effect from a population of topics at well below the nominal rate. For a plausible scenario, the confidence interval from a single-topic study might only be half as wide as an interval that captures the typical effect at the nominal rate. We highlight three important conclusions. First, we emphasize that researchers and readers must take care when generalizing the inferences from single-topic studies to a larger population of possible studies. Second, we demonstrate the critical importance of similarity across topics in drawing inferences and encourage researchers to consider designs that explicitly estimate and leverage similarity. Third, we emphasize that, despite their limitations, single-topic experiments have some important advantages.

This commentary addresses the modeling and final analytical path taken, as well as the terminology used, in the paper “Hierarchical diagnostic classification models: a family of models for estimating and testing attribute hierarchies” by Templin and Bradshaw (Psychometrika, doi:10.1007/s11336-013-9362-0, 2013). It raises several issues concerning use of cognitive diagnostic models that either assume attribute hierarchies or assume a certain form of attribute interactions. The issues raised are illustrated with examples, and references are provided for further examination.

Multinomial processing tree models are widely used in many areas of psychology. Their application relies on the assumption of parameter homogeneity, that is, on the assumption that participants do not differ in their parameter values. Tests for parameter homogeneity are proposed that can be routinely used as part of multinomial model analyses to defend the assumption. If parameter homogeneity is found to be violated, a new family of models, termed latent-class multinomial processing tree models, can be applied that accommodates parameter heterogeneity and correlated parameters, yet preserves most of the advantages of the traditional multinomial method. Estimation, goodness-of-fit tests, and tests of other hypotheses of interest are considered for the new family of models.

Multinomial processing tree models are widely used in many areas of psychology. A hierarchical extension of the model class is proposed, using a multivariate normal distribution of person-level parameters with the mean and covariance matrix to be estimated from the data. The hierarchical model allows one to take variability between persons into account and to assess parameter correlations. The model is estimated using Bayesian methods with weakly informative hyperprior distribution and a Gibbs sampler based on two steps of data augmentation. Estimation, model checks, and hypotheses tests are discussed. The new method is illustrated using a real data set, and its performance is evaluated in a simulation study.

The theory of signal detection is convenient for measuring mnemonic ability in recognition memory paradigms. In these paradigms, randomly selected participants are asked to study randomly selected items. In practice, researchers aggregate data across items or participants or both. The signal detection model is nonlinear; consequently, analysis with aggregated data is not consistent. In fact, mnemonic ability is underestimated, even in the large-sample limit. We present two hierarchical Bayesian models that simultaneously account for participant and item variability. We show how these models provide for accurate estimation of participants’ mnemonic ability as well as the memorability of items. The model is benchmarked with a simulation study and applied to a novel data set.

Hierarchical models are often considered to measure latent concepts defining nested sets of manifest variables. Therefore, by supposing a hierarchical relationship among manifest variables, the general latent concept can be represented by a tree structure where each internal node represents a specific order of abstraction for the latent concept measured. In this paper, we propose a new latent factor model called second-order disjoint factor analysis in order to model an unknown hierarchical structure of the manifest variables with two orders. This is a second-order factor analysis, which—respect to the second-order confirmatory factor analysis—is exploratory, nested and estimated simultaneously by maximum likelihood method. Each subset of manifest variables is modeled to be internally consistent and reliable, that is, manifest variables related to a factor measure “consistently” a unique theoretical construct. This feature implies that manifest variables are positively correlated with the related factor and, therefore, the associated factor loadings are constrained to be nonnegative. A cyclic block coordinate descent algorithm is proposed to maximize the likelihood. We present a simulation study that investigates the ability to get reliable factors. Furthermore, the new model is applied to identify the underlying factors of well-being showing the characteristics of the new methodology. A final discussion completes the paper.

The present article proposes and evaluates marginal maximum likelihood (ML) estimation methods for hierarchical multinomial processing tree (MPT) models with random and fixed effects. We assume that an identifiable MPT model with S parameters holds for each participant. Of these S parameters, R parameters are assumed to vary randomly between participants, and the remaining $S-R$\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$S-R$$\end{document} parameters are assumed to be fixed. We also propose an extended version of the model that includes effects of covariates on MPT model parameters. Because the likelihood functions of both versions of the model are too complex to be tractable, we propose three numerical methods to approximate the integrals that occur in the likelihood function, namely, the Laplace approximation (LA), adaptive Gauss–Hermite quadrature (AGHQ), and Quasi Monte Carlo (QMC) integration. We compare these three methods in a simulation study and show that AGHQ performs well in terms of both bias and coverage rate. QMC also performs well but the number of responses per participant must be sufficiently large. In contrast, LA fails quite often due to undefined standard errors. We also suggest ML-based methods to test the goodness of fit and to compare models taking model complexity into account. The article closes with an illustrative empirical application and an outlook on possible extensions and future applications of the proposed ML approach.

A unifying framework for generalized multilevel structural equation modeling is introduced. The models in the framework, called generalized linear latent and mixed models (GLLAMM), combine features of generalized linear mixed models (GLMM) and structural equation models (SEM) and consist of a response model and a structural model for the latent variables. The response model generalizes GLMMs to incorporate factor structures in addition to random intercepts and coefficients. As in GLMMs, the data can have an arbitrary number of levels and can be highly unbalanced with different numbers of lower-level units in the higher-level units and missing data. A wide range of response processes can be modeled including ordered and unordered categorical responses, counts, and responses of mixed types. The structural model is similar to the structural part of a SEM except that it may include latent and observed variables varying at different levels. For example, unit-level latent variables (factors or random coefficients) can be regressed on cluster-level latent variables. Special cases of this framework are explored and data from the British Social Attitudes Survey are used for illustration. Maximum likelihood estimation and empirical Bayes latent score prediction within the GLLAMM framework can be performed using adaptive quadrature in gllamm, a freely available program running in Stata.

This paper provides a statistical framework for estimating higher-order characteristics of the response time distribution, such as the scale (variability) and shape. Consideration of these higher order characteristics often provides for more rigorous theory development in cognitive and perceptual psychology (e.g., Luce, 1986). RT distribution for a single participant depends on certain participant characteristics, which in turn can be thought of as arising from a distribution of latent variables. The present work focuses on the three-parameter Weibull distribution, with parameters for shape, scale, and shift (initial value). Bayesian estimation in a hierarchical framework is conceptually straightforward. Parameter estimates, both for participant quantities and population parameters, are obtained through Markov Chain Monte Carlo methods. The methods are illustrated with an application to response time data in an absolute identification task. The behavior of the Bayes estimates are compared to maximum likelihood (ML) estimates through Monte Carlo simulations. For small sample size, there is an occasional tendency for the ML estimates to be unreasonably extreme. In contrast, by borrowing strength across participants, Bayes estimation “shrinks” extreme estimates. The results are that the Bayes estimators are more accurate than the corresponding ML estimators.

When researchers design an experiment, they usually hold potentially relevant features of the experiment constant. We call these details the “topic” of the experiment. For example, researchers studying the impact of party cues on attitudes must inform respondents of the parties’ positions on a particular policy. In doing so, researchers implement just one of many possible designs . Clifford, Leeper, and Rainey (2023. “Generalizing Survey Experiments Using Topic Sampling: An Application to Party Cues.” Forthcoming in Political Behavior. https://doi.org/10.1007/s11109-023-09870-1) argue that researchers should implement many of the possible designs in parallel—what they call “topic sampling”—to generalize to a larger population of topics. We describe two estimators for topic-sampling designs: First, we describe a nonparametric estimator of the typical effect that is unbiased under the assumptions of the design; and second, we describe a hierarchical model that researchers can use to describe the heterogeneity. We suggest describing the heterogeneity across topics in three ways: (1) the standard deviation in treatment effects across topics, (2) the treatment effects for particular topics, and (3) how the treatment effects for particular topics vary with topic-level predictors. We evaluate the performance of the hierarchical model using the Strengthening Democracy Challenge megastudy and show that the hierarchical model works well.

When working with grouped data, investigators may choose between “fixed effects” models (FE) with specialized (e.g., cluster-robust) standard errors, or “multilevel models” (MLMs) employing “random effects.” We review the claims given in published works regarding this choice, then clarify how these approaches work and compare by showing that: (i) random effects employed in MLMs are simply “regularized” fixed effects; (ii) unmodified MLMs are consequently susceptible to bias—but there is a longstanding remedy; and (iii) the “default” MLM standard errors rely on narrow assumptions that can lead to undercoverage in many settings. Our review of over 100 papers using MLM in political science, education, and sociology show that these “known” concerns have been widely ignored in practice. We describe how to debias MLM’s coefficient estimates, and provide an option to more flexibly estimate their standard errors. Most illuminating, once MLMs are adjusted in these two ways the point estimate and standard error for the target coefficient are exactly equal to those of the analogous FE model with cluster-robust standard errors. For investigators working with observational data and who are interested only in inference on the target coefficient, either approach is equally appropriate and preferable to uncorrected MLM.

This article examines the determinants of the annual overall spending levels of political parties from a comparative perspective. On the basis of a dataset with 1,317 observations from 99 parties in nine European parliamentary democracies, we illustrate that the spending levels of parties – calculated on the basis of Nassmacher’s index of political spending – have not systematically increased over the past decade. A multilevel analysis shows that, at the country level, spending levels are higher in countries with a longer tradition of public funding, a higher effective number of parties and a shorter democratic tradition. They are also higher in election years, but this effect is moderated when campaign spending limits apply. At the party level, spending levels increase with party strength and party age. Party ideology and government participation, on the other hand, do not have an effect.

Objectives: The aim of this study was to propose a statistical model that takes into account clinical data on earlier versions when evaluating the latest version of an implantable medical device (IMD).

Methods: We compared the performances of a Bayesian three-level hierarchical meta-analysis model with those of a Bayesian random-effects model through a simulation study. Posterior mean estimates of the success rate for each IMD version were computed as well as the probability that the latest version improved in effectiveness. Models were compared using the Deviance Information Criterion (DIC), the estimated bias and the standard deviation of the mean success rates. Sensitivity analyses to the choice of the priors were performed. These methods were applied to the evaluation of an intracranial stent used to treat wide-necked aneurysms.

Results: When IMD versions did not differ in effectiveness, the best-fitting model was the random-effects model. By contrast, when there was a version effect, the hierarchical model was selected in more than 95 percent of the cases. It provided precise estimations of success rates of each IMD version and allowed detecting an improvement in effectiveness of the latest version, with a low influence of the choice of the priors. No evidence of benefit from the latest version of the intracranial stent was found.

Conclusions: In the setting of IMD assessment, comparison of DIC between the two proposed models appeared useful for detecting version effects. In that case, Bayesian hierarchical meta-analysis model may help the decision maker by providing useful information on the latest version of IMD compared with the previous versions.

Category-specific impairments of object recognition and naming are among the most intriguing disorders in neuropsychology, affecting the retrieval of knowledge about either living or nonliving things. They can give us insight into the nature of our representations of objects: Have we evolved different neural systems for recognizing different categories of object? What kinds of knowledge are important for recognizing particular objects? How does visual similarity within a category influence object recognition and representation? What is the nature of our semantic knowledge about different objects? We review the evidence on category-specific impairments, arguing that deficits even for one class of object (e.g., living things) cannot be accounted for in terms of a single information processing disorder across all patients; problems arise at contrasting loci in different patients. The same apparent pattern of impairment can be produced by damage to different loci. According to a new processing framework for object recognition and naming, the hierarchical interactive theory (HIT), we have a hierarchy of highly interactive stored representations. HIT explains the variety of patients in terms of (1) lesions at different levels of processing and (2) different forms of stored knowledge used both for particular tasks and for particular categories of object.