We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the “proportional reduction in loss” (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.

We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.

Coefficient κ is generally defined in terms of procedures of computation rather than in terms of a population. Here a population definition is proposed. On this basis, the interpretation of κ as a measure of diagnostic reliability in characterizing an individual, and the effect of reliability, as measured by κ, on estimation bias, precision, and test power are examined. Factors influencing the magnitude of κ are identified. Strategies to improve reliability are proposed, including that of combining multiple unreliable diagnoses.

Studies of agreement commonly occur in psychiatric research. For example, researchers are often interested in the agreement among radiologists in their review of brain scans of elderly patients with dementia or in the agreement among multiple informant reports of psychopathology in children. In this paper, we consider the agreement between two raters when rating a dichotomous outcome (e.g., presence or absence of psychopathology). In particular, we consider logistic regression models that allow agreement to depend on both rater- and subject-level covariates. Logistic regression has been proposed as a simple method for identifying covariates that are predictive of agreement (Coughlin et al., 1992). However, this approach is problematic since it does not take account of agreement due to chance alone. As a result, a spurious association between the probability (or odds) of agreement and a covariate could arise due entirely to chance agreement. That is, if the prevalence of the dichotomous outcome varies among subgroups of the population, then covariates that identify the subgroups may appear to be predictive of agreement. In this paper we propose a modification to the standard logistic regression model in order to take proper account of chance agreement. An attractive feature of the proposed method is that it can be easily implemented using existing statistical software for logistic regression. The proposed method is motivated by data from the Connecticut Child Study (Zahner et al., 1992) on the agreement among parent and teacher reports of psychopathology in children. In this study, parents and teachers provide dichotomous assessments of a child's psychopathology and it is of interest to examine whether agreement among the parent and teacher reports is related to the age and gender of the child and to the time elapsed between parent and teacher assessments of the child.

Cohen's kappa index is reformulated for multiple classifications based on exchangeable random variables. It is found that kappa is between 0 and 1 inclusive. Two characterizations for kappa are stated in terms of the relationship between such random variables. Within the normal test score model, kappa increases with test reliability and test length. Furthermore, when based on binary classifications, kappa is an inverse U-shaped function of the cutoff score. These trends also hold for the beta-binomial test score model.

Experiments are considered where each of a sample of subjects is assigned to one of C categories separately by each of a fixed or varying group of observers. Building on earlier publications, general procedures are proposed to analyze agreements and disagreements among observers. In the case of a varying group of observers, it is shown that it is not necessary to demand a constant number of observers per subject. In the case of a fixed group of observers, the problem of missing data is considered.

The procedures are illustrated within the context of two clinical diagnosis examples. In the first example it is investigated which categories are relatively hard to distinguish from one another; a new theorem is applied that shows a useful property of the statistic kappa. In the second example it is investigated if a subgroup of observers can be found with a significantly higher degree of interobserver agreement.

False broomweed is a troublesome weed on south Texas rangelands. The plant suppresses the growth of desirable herbaceous plant species and is unpalatable to livestock and wildlife. The objectives of this study were to evaluate multispectral satellite data for automated detection, classification, and mapping of false broomweed infestations. Determining the optimum phenological conditions for false broomweed detection was a major goal. Results indicate that satellite data can be used to detect major stands of this shrub and map the relative extent of infested areas. The best classification was obtained when the foliage of the shrub was fully developed and during periods of low herbaceous biomass production. Limited ground or aerial surveying will be needed to produce more exact estimates of the extent of false broomweed stands; however, these efforts can be focused on areas identified by satellite classification.

Interrater reliability of the Hamilton Rating Scale for Depression

Interrater reliability of the Dutch version of the Hamilton Rating Scale for Depression (HRSD 17 items) was investigated. Although sufficient, a kappa of .65, found for all raters, was not very high. Specifically item 2 (feelings of guilt) and item 11 (psychic anxiety) seemed difficult to judge. The most experienced raters achieved the highest kappas as well on the total score of the HRSD.

It is concluded that this version of the HRSD, if administered by experienced raters, is a reasonably reliable instrument to obtain a certain ‘cut-off score (for instance in-exclusion criterium) as well as to measure the change in severity of the depression during treatment.