Observer Bias Affects the Corrections to the Faulty Observations

  Observer bias can affect the correction to the estimates of the hidden probabilities for the occurrence of good and bad items. In the last blog the observer assessments were unbiased for both examples. In the first example below the first observer is less likely to make an error in identifying good items while the second observer makes fewer mistakes on bad items. The estimated probabilities ends up slightly biased in favor of good items since there are more good items than bad in the sample. In the second example below both observers more accurately identify bad items and the estimated probabilities have a slight shift towards the occurrence of bad items.

Since there were 100 items in each sample one would expect the rms error in the estimated mean probabilities to be about 1/√100=0.1 times the rms error in the 100 items in the sample so the mean probability estimates should be accurate to about 3 digits.