Reliability (separation index) means "reproducibility of relative measure location". It does not report on the quality of the data. So "high reliability" (of persons or items) means that there is a high probability that persons (or items) estimated with high measures actually do have higher measures than persons (or items) estimated with low measures. If you want high reliability, you need a wide sample and/or low measurement error. So, if you want high person (test) reliability, you need a person sample with a large ability (or whatever) range and/or an instrument with many items (or long rating scales). If you want high item reliability, you need a test with a large item difficulty range and/or a large sample of persons. Usually low item reliability is because the person sample size is too small to establish a reproducible item difficulty hierarchy.

 

Missing data: if some persons have missing observations, these can considerably reduce precision, and so lower reliability estimates.  Suggestion: omit person-records with missing data when estimating reliabilities.

 

Person (sample, test) reliability depends chiefly on

1) Sample ability variance. Wider ability range = higher person reliability.

2) Length of test (and rating scale length). Longer test = higher person reliability

3) Number of categories per item. More categories = higher person reliability

4) Sample-item targeting. Better targeting = higher person reliability

It is independent of sample size. It is largely uninfluenced by model fit.

 

Item reliability depends chiefly on

1) Item difficulty variance. Wide difficulty range = high item reliability

2) Person sample size. Large sample = high item reliability

It is independent of test length. It is largely uninfluenced by model fit.

 

Tentative guidelines:

Person reliability: Does your test discriminate the sample into enough levels for your purpose? 0.9 = 3 or 4 levels. 0.8 = 2 or 3 levels. 0.5 = 1 or 2 levels.

Item reliability: Low reliability means that your sample is not big enough to precisely locate the items on the latent variable.

Rater reliability: Low "separation" reliability is better, because we want raters to be reliably the same, not reliably different.

 

The Winsteps "person reliability" is equivalent to the traditional "test" reliability. Low values indicate a narrow range of person measures, or a small number of items. To increase person reliability, test persons with more extreme abilities (high and low), lengthen the test. Improving the test targeting may help slightly.

 

The Winsteps "item reliability" has no traditional equivalent. Low values indicate a narrow range of item measures, or a small sample. To increase "item reliability", test more people. In general, low item reliability means that your sample size is too small for stable item estimates based on the current data. If you have anchored values, then it is the item reliability of the source from which the anchor values emanate which is crucial, not the current sample.

 

The "model" person reliability (including measures for extreme scores) is an upper bound to this value, when persons are ordered by measures.

 

The "real" person reliability (including measures for extreme scores) is a lower bound to this value, when persons are ordered by measures

 

The traditional "test reliability", as defined by Charles Spearman in 1904, etc., is the "true person variance / observed person variance" for this sample on these test items. So it is really a "person sample reliability" rather than a "test reliability", where reliability = reproducibility of person ordering. The "true person variance" cannot be known, but it can be approximated. KR-20 approximates it by summarizing item point-biserials. Cronbach Alpha approximates it with an analysis of variance. Winsteps approximates it using the measure standard errors.

 

The separation coefficient and reliability computations are computed with and without any elements with extreme measures. Since the measures for extreme scores are imprecise, reliability statistics which include extreme scores are often lower than their non-extreme equivalents. Conventional computation of a reliability coefficient (KR-20, Cronbach Alpha) includes persons with extreme scores. The classical reliability computation includes extreme scores (if any) is the conventional reliability, and usually produces an estimate between the MODEL and REAL values, closer to the MODEL or even above it.

 

KR-20 value is an estimate of the value when persons are ordered by raw scores. CRONBACH ALPHA (KR-20) KID RAW SCORE RELIABILITY is the conventional "test" reliability index. It reports an approximate test reliability based on the raw scores of this sample. It is only reported for complete data. An apparent paradox is that extreme scores have perfect precision, but extreme measures have perfect imprecision.

 

Winsteps computes upper and lower boundary values for the True Reliability. The lower boundary is the Real Reliability. The upper boundary is the Model Reliability. The unknowable True Reliability lies somewhere between these two. As contradictory sources of noise are remove from the data, the True Reliability approaches the Model Reliability

 

Conventionally, only a Person ("Test") Reliability is reported. The relationship between raw-score-based reliability (i.e., KR-20, Cronbach Alpha) and measure-based reliability is complex, see www.rasch.org/rmt/rmt113l.htm - in general, Cronbach Alpha overestimates reliability, Rasch underestimates it. So, when it is likely that the Rasch reliability will be compared with conventional KR-20 or Cronbach Alpha reliabilities (which are always computed assuming the data match their assumptions), then include extreme persons and report the higher Rasch reliability, the "Model" reliability, computed on the assumption that all unexpectedness in the data is in accord with Rasch model predictions. The big differences between Score and Measure reliabilities occur when

(a) there are extreme scores. These increase score reliability, but decrease measure reliability.

(b) missing data. Missing data always decreases measure reliability. If the missing data are imputed at their expected values (in order to make conventional reliability formulas computable), they increase score reliability. Winsteps attempts to adjust the raw-score reliability for this inflation in the raw-score reliability, but can only do the adjustment in an approximate way.

 

Winsteps also reports an item reliability, "true item variance / observed item variance". When this value is low, it indicates that the sample size may be too small for stable comparisons between items.

 

Anchored values are treated as though they are the "true values" of the MLE estimates. Their local standard errors are estimated using the current data in the same way as unanchored MLE standard error estimates. It is the measures (anchored or unanchored) and local standard errors that are used in the reliability computations. If you wish to compute reliabilities using different standard error estimates (e.g., the ones when the anchor values were generated), then please perform a separate reliability computation (using Excel).

 

You can easily check the Winsteps reliability estimate computation yourself.

 

Read the Winsteps PFILE= into an Excel spreadsheet.

 

Compute the STDEVP standard deviation of the person measures. Square it. This is the "Observed variance".

 

"Model" Reliability: Take the standard ERROR column. Square each entry. Sum the squared entries. Divide that sum by the count of entries. This is the "Model Error variance" estimate. Then,

Model Reliability = True Variance / Observed Variance = (Observed Variance - Model Error Variance) / Observed Variance.

 

"Real" Reliability: Take the standard ERROR column. Square each entry, SE². In another column, put SE²*Maximum [1.0, INFIT mean-square). Divide that sum by the count of entries. This is the "Real Error variance" estimate. Then,

Real Reliability = True Variance / Observed Variance = (Observed Variance - Real Error Variance) / Observed Variance.

 

---

 

Separation, Strata and Reliability

 

The crucial elements in the computation of reliability are the "True" variance and the Error variance. These are squared distances and so difficulty to conceptualize directly. It is easier to think of their square-roots, the "True" standard deviation (TSD) and the root-mean-square standard error (RMSE).

 

SEPARATION coefficient is the ratio of the PERSON (or ITEM)TRUE S.D., the "true" standard deviation, to RMSE, the error standard deviation. It provides a ratio measure of separation in RMSE units, which is easier to interpret than the reliability correlation. This is analogous to the Fisher Discriminant Ratio. SEPARATION coefficient² is the signal-to-noise ratio, the ratio of "true" variance to error variance.

 

RELIABILITY (separation index) is a separation reliability. The PERSON (or ITEM) reliability is equivalent to KR-20, Cronbach Alpha, and the Generalizability Coefficient. The relationship between SEPARATION coefficent and RELIABILITY (separation index) is

RELIABILITY = SEPARATION coefficient²/(1+SEPARATION coefficient²)

or SEPARATION coefficent = square-root(RELIABILITY/(1-RELIABILITY)).

 

Separation (if the outlying measures are accidental) or Strata (if the outlying measures represent true performances). These numbers are statistical abstractions, but there empirical meaning is indicated by locating the Separation or Strata levels in the observed distribution at (3 * "Observed S.D." / Separation) units apart, centered on the sample mean.

 

 

There is more at www.rasch.org/rmt/rmt94n.htm and www.rasch.org/rmt/rmt163f.htm

 

---

 

Spearman-Brown Prophecy Formula

 

How many items (or persons) are required to produce the reliability I want with the sample of persons and the same type of items (or this test and the same type of persons)?

 

T = target number of items, RT = target person reliability

C = current number of items, RC = current person reliability

 

1. Predict number of persons = T = C * RT * (1-RC) / ( (1-RT) * RC)

 

Example: the current test length is C = 10 items, and the current person reliability is RC = 0.3. We want a person reliability of RT = 0.8.

Target number of items is T = 10 * 0.8 * (1-0.3) / ( (1-0.8)* 0.3) = 94 items.

 

2. Predicted reliability = RT = T * RC / ( C * (1-RC) + T * RC)

 

Example: we have a test of C = 11 items of person reliability RC = 0.5, what is the predicted reliability of a test of T = 17 items?

Predicted reliability RT =  17 * 0.5 / ( 11 * (1-0.5) + 17 * 0.5) = 0.61

 

---

 

Test-Retest Reliability

 

is the correlation between the person measures obtained from two administrations of the same test to the same persons. The expected value of the test-retest reliability is the person reliability of the first administration.

 

The "Smallest Detectable Difference" = "Smallest statistically significant difference in a person's measures" when a test is administered twice to a person under normal conditions = 1.96*(person sample S.D.)*(test-retest reliability).