Using Concurrence Counts for Comparisons

  In the last couple of blogs I tried to show how the expected number of good items in a sample can be estimated if the agreement and disagreement of two testers were known for correct assessments on the same set of items. One just needs the counts of the concurrence since from them one can determine what the testers observations were as indicated in this diagram. The counts in the last column are just the sum of the counts for the two path leading to the combined counts.

If one tries to make an estimate N_G using the actual observations one ends up with an estimate of N₀, the number of items in the sample instead.

So the testers' assessments themselves don't help us very much. In general we need a table of the concurrence of the actual number of good and bad items. We can alter the problem by asking how well a pass/fail function test will predict whether an item will function for a specified length of time with those that do being the number of "good" items. So we need to consider diagrams like these for the two testers.

After the true counts of concurrence have been determined on can use them to get the conditional probabilities for the testers which in turn allow us to evaluate the testers and their tests.

Supplemental (Oct 4): The reference to a concurrence matrix above would be more properly be called a concurrence table. The conditional probabilities are part of a matrix since they allow one set to be used to compute the other if one set of probabilities is known. The reason for the failure of the testers' concurrence estimate is due to the presence of false positives in the counts used.