The paper provides a survey of 18 years’ progress that my colleagues, students (both former and current) and I made in a prominent research area in Psychometrics—Computerized Adaptive Testing (CAT). We start with a historical review of the establishment of a large sample foundation for CAT. It is worth noting that the asymptotic results were derived under the framework of Martingale Theory, a very theoretical perspective of Probability Theory, which may seem unrelated to educational and psychological testing. In addition, we address a number of issues that emerged from large scale implementation and show that how theoretical works can be helpful to solve the problems. Finally, we propose that CAT technology can be very useful to support individualized instruction on a mass scale. We show that even paper and pencil based tests can be made adaptive to support classroom teaching.

Replenishing item pools for on-line ability testing requires innovative and efficient data collection designs. By generating local D-optimal designs for selecting individual examinees, and consistently estimating item parameters in the presence of error in the design points, sequential procedures are efficient for on-line item calibration. The estimating error in the on-line ability values is accounted for with an item parameter estimate studied by Stefanski and Carroll. Locally D-optimal n-point designs are derived using the branch-and-bound algorithm of Welch. In simulations, the overall sequential designs appear to be considerably more efficient than random seeding of items.

In this paper we propose a class of sequential urn designs based on generalized Pólya urn (GPU) models for balancing the allocations of two treatments in sequential clinical trials. In particular, we consider a GPU model characterized by a 2 x 2 random addition matrix with null balance (i.e. null row sums) and replacement rule depending upon the urn composition. Under this scheme, the urn process has a Markovian structure and can be regarded as a random extension of the classical Ehrenfest model. We establish almost sure convergence and asymptotic normality for the frequency of treatment allocations and show that in some peculiar cases the asymptotic variance of the design admits a natural representation based on the set of orthogonal polynomials associated with the corresponding Markov process.

We consider a controlled i.i.d. process, where several i.i.d. sources are sampled sequentially. At each time instant, a controller determines from which source to obtain the next sample. Any causal sampling policy, possibly history-dependent, may be employed. The purpose is to characterize the extremal large deviations of the sample mean, namely to obtain asymptotic rate bounds (similar to and extending Cramér's theorem) which hold uniformly over all sampling policies. Lower and upper bounds are obtained, and it is shown that in many (but not all) cases stationary sampling policies are sufficient to obtain the extremal large deviations rates. These results are applied to a hypothesis testing problem, where data samples may be sequentially chosen from several i.i.d. sources (representing different types of experiments). The analysis provides asymptotic estimates for the error probabilities, corresponding both to optimal and to worst-case sampling policies.

In a series of papers Whittle has proposed locating the optimal boundary between the continuation region and the stopping region, in the no-overshoot approximation, for a class of sequential analysis problems by using the criterion that the gradient of the objective function must be continuous at the boundary. It is shown here that the result originally proposed by Whittle is not true in the generality which he supposed, and an indication is given of some of the circumstances in which it does hold.