Relationships between the Rasch model and both the law of comparative judgment and additive conjoint measurement are discussed. The distance between the ability of Person a and the difficult of Item i is, in the Rasch model, the baseline value corresponding to the probability that a will respond correctly to i, where this probability is interpreted as the area under a logistic curve (which is substantially equivalent to the normal curve) and is thus an application of the law of comparative judgment. Under certain assumptions, the Rasch model is also a special case of additive conjoint measurement and, properly reinterpreted, may be usefully applied in contexts other than individual differences.

A method of estimating item characteristic functions is proposed, in which a set of test items, whose operating characteristics are known and which give a constant test information function for a substantially wide range of ability, are used. The method is based on the maximum likelihood estimates of ability for a group of several hundred examinees. Throughout the present study the Monte Carlo method is used.

In this paper, I will review some aspects of psychometric projects that I have been involved in, emphasizing the nature of the work of the psychometricians involved, especially the balance between the statistical and scientific elements of that work. The intent is to seek to understand where psychometrics, as a discipline, has been and where it might be headed, in part at least, by considering one particular journey (my own). In contemplating this, I also look to psychometrics journals to see how psychometricians represent themselves to themselves, and in a complementary way, look to substantive journals to see how psychometrics is represented there (or perhaps, not represented, as the case may be). I present a series of questions in order to consider the issue of what are the appropriate foci of the psychometric discipline. As an example, I present one recent project at the end, where the roles of the psychometricians and the substantive researchers have had to become intertwined in order to make satisfactory progress. In the conclusion I discuss the consequences of such a view for the future of psychometrics.

A latent trait model is presented for the repeated measurement of ability based on a multidimensional conceptualization of the change process. A simplex structure is postulated to link item performance under a given measurement condition or occasion to initial ability and to one or more modifiabilities that represent individual differences in change. Since item discriminations are constrained to be equal within a measurement condition, the model belongs to the family of multidimensional Rasch models. Maximum likelihood estimators of the item parameters and abilities are derived, and an example provided that shows good recovery of both item and ability parameters. Properties of the model are explored, particularly for several classical issues in measuring change.

This paper traces the course of the consequences of viewing test responses as simply providing dichotomous data concerning ordinal relations. It begins by proposing that the score matrix is best considered to be items-plus-persons by items-plus-persons, and recording the wrongs as well as the rights. This shows how an underlying order is defined, and was used to provide the basis for a tailored testing procedure. It also was used to define a number of measures of test consistency. Test items provide person dominance relations, and the relations provided by one item can be in one of three relations with a second one: redundant, contradictory, or unique. Summary statistics concerning the number of relations of each kind are easy to get and provide useful information about the test, information which is related to but different from the usual statistics. These concepts can be extended to form the basis of a test theory which is based on ordinal statistics and frequency counts and which invokes the concept of true scores only in a limited sense.

This paper presents a contribution to the sampling theory of a set of homogeneous tests which differ only in length, test length being regarded as an essential test parameter. Observed variance-covariance matrices of such measurements are taken to follow a Wishart distribution. The familiar true score-and-error concept of classical test theory is employed. Upon formulation of the basic model it is shown that in a combination of such tests forming a “total” test, the singal-to-noise ratio of the components is additive and that the inverse of the population variance-covariance matrix of the component measures has all of its off-diagonal elements equal, regardless of distributional assumptions. This fact facilitates the subsequent derivation of a statistical sampling theory, there being at most m + 1 free parameters when m is the number of component tests. In developing the theory, the cases of known and unknown test lengths are treated separately. For both cases maximum-likelihood estimators of the relevant parameters are derived. It is argued that the resulting formulas will remain reasonable even if the distributional assumptions are too narrow. Under these assumptions, however, maximum-likelihood ratio tests of the validity of the model and of hypotheses concerning reliability and standard error of measurement of the total test are given. It is shown in each case that the maximum-likelihood equations possess precisely one acceptable solution under rather natural conditions. Application of the methods can be effected without the use of a computer. Two numerical examples are appended by way of illustration.

A test theory using only ordinal assumptions is presented. It is based on the idea that the test items are a sample from a universe of items. The sum across items of the ordinal relations for a pair of persons on the universe items is analogous to a true score. Using concepts from ordinal multiple regression, it is possible to estimate the tau correlations of test items with the universe order from the taus among the test items. These in turn permit the estimation of the tau of total score with the universe. It is also possible to estimate the odds that the direction of a given observed score difference is the same as that of the true score difference. The estimates of the correlations between items and universe and between total score and universe are found to agree well with the actual values in both real and artificial data.

We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the “proportional reduction in loss” (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.

We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.

We give an account of Classical Test Theory (CTT) in terms of the more fundamental ideas of Item Response Theory (IRT). This approach views classical test theory as a very general version of IRT, and the commonly used IRT models as detailed elaborations of CTT for special purposes. We then use this approach to CTT to derive some general results regarding the prediction of the true-score of a test from an observed score on that test as well from an observed score on a different test. This leads us to a new view of linking tests that were not developed to be linked to each other. In addition we propose true-score prediction analogues of the Dorans and Holland measures of the population sensitivity of test linking functions. We illustrate the accuracy of the first-order theory using simulated data from the Rasch model, and illustrate the effect of population differences using a set of real data.

There is a unity underlying the diversity of models for the analysis of multivariate data. Essentially, they constitute a family models, most generally nonlinear, for structural/functional relations between variables drawn from a behavior domain.

There are three fundamental problems in Sijtsma (Psychometrika, 2008): (1) contrary to the name, the glb is not the greatest lower bound of reliability but rather is systematically less than ωt (McDonald, Test theory: A unified treatment, Erlbaum, Hillsdale, 1999), (2) we agree with Sijtsma that when considering how well a test measures one concept, α is not appropriate, but recommend ωt rather than the glb, and (3) the end user needs procedures that are readily available in open source software.

It is shown that IRTs information function for an item is functionally related to “local” versions of classical test theories' signal/noise ratio and reliability coefficient.

This paper argues that test data are ordinal, that latent trait scores are only determined ordinally, and that test data are used largely for ordinal purposes. Therefore it is desirable to develop a test theory based only on ordinal assumptions. A set of ordinal assumptions is presented, including an ordinal version of local independence. From these assumptions it is first shown that the gamma-correlation between two tests is the product of their gamma-correlations with the true latent order. The theory is generalized to allow for heterogeneous tests by defining a weighted average local independence. The tau-correlations between total score and the latent order can be found in both homogeneous and heterogeneous cases, and a system of differential item weighting to maximize the tau-correlation between weighted items and the latent order is provided. Thus a purely ordinal test theory seems possible.

Two simple classes of mastery scores which are suitable for hand calculations are presented for beta-binomial test score distributions combined with linear and cubic referral success. The models provide a simple way to explore the consequences of selecting an arbitrary mastery score. Such assessment would be useful whenever the test user is not willing to post a priori a loss ratio, but wishes to look at the various consequences before aiming at a particular score.

Educational assessment concerns inference about students' knowledge, skills, and accomplishments. Because data are never so comprehensive and unequivocal as to ensure certitude, test theory evolved in part to address questions of weight, coverage, and import of data. The resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. Issues of evidence and inference in educational assessment are discussed from this perspective.

Cohen's kappa index is reformulated for multiple classifications based on exchangeable random variables. It is found that kappa is between 0 and 1 inclusive. Two characterizations for kappa are stated in terms of the relationship between such random variables. Within the normal test score model, kappa increases with test reliability and test length. Furthermore, when based on binary classifications, kappa is an inverse U-shaped function of the cutoff score. These trends also hold for the beta-binomial test score model.

A general model is presented for homogeneous, dichotomous items when the answer key is not known a priori. The model is structurally related to the two-class latent structure model with the roles of respondents and items interchanged. For very small sets of respondents, iterative maximum likelihood estimates of the parameters can be obtained by existing methods. For other situations, new estimation methods are developed and assessed with Monte Carlo data. The answer key can be accurately reconstructed with relatively small sets of respondents. The model is useful when a researcher wants to study objectively the knowledge possessed by members of a culturally coherent group that the researcher is not a member of.

A chain of lower-bound inequalities leading to the greatest lower bound to reliability is established for the internal consistency of a composite of unit-weighted components. The chain includes the maximum split-half coefficient, the lowest coefficient consistent with nonimaginary common factors, and the lowest coefficient consistent with nonimaginary common and unique factors. Optimization theory is utilized to determine the conditions that are requisite for the inequalities. Convergence proofs demonstrate that the coefficients can be attained. Rapid algorithms obtain estimates of the coefficients with sample data. The theory yields methods for splitting items into maximally similar sets and for exploratory factor analysis based on a theoretical solution to the communality problem.

A number of models for categorical item response data have been proposed in recent years. The models appear to be quite different. However, they may usefully be organized as members of only three distinct classes, within which the models are distinguished only by assumptions and constraints on their parameters. “Difference models” are appropriate for ordered responses, “divide-by-total” models may be used for either ordered or nominal responses, and “left-side added” models are used for multiple-choice responses with guessing. The details of the taxonomy and the models are described in this paper.