A Commentary on Bayesian Games etc.

  The last few blogs dealt with problems related to Bayesian inference. The chief obstacle to such an analysis is getting a good estimate of the conditional probabilities when the reliability of the observers are unknown. We found that using two observers can give an improved estimate of the expected values of the likely unknown counts which are needed to determine the conditional probabilities.

The formulas for the hidden probabilities require counts of the number of events that the observers agree on. Rutherford gives the formula for computing the probability of two simultaneous events P and Q occurring but his papers don't indicate that he used these formulas to get improved estimates of rates based on the observations of two observers. Fuller's proofreading problem doesn't indicate a method of solution either. I have a copy of Fuller book (3rd Ed. see p. 170) and derived the formulas for the solution to the problem on my own. The proofreading problem and formulas can be found in Ross, Introduction to Probability and Statistics for Engineers and Scientists, p. 234f.

A related problem in Bayesian inference would be the determination of the likely number of false positives and false negatives for the observers using a particular method for assessments. If the conditional probabilities are known one can do so. A problem with statistical analysis is that rare events tend to be excluded from observations. Another problem is extending restricted studies to more general cases where the rates are likely not to be the same.

Observer Bias Affects the Corrections to the Faulty Observations

  Observer bias can affect the correction to the estimates of the hidden probabilities for the occurrence of good and bad items. In the last blog the observer assessments were unbiased for both examples. In the first example below the first observer is less likely to make an error in identifying good items while the second observer makes fewer mistakes on bad items. The estimated probabilities ends up slightly biased in favor of good items since there are more good items than bad in the sample. In the second example below both observers more accurately identify bad items and the estimated probabilities have a slight shift towards the occurrence of bad items.

Since there were 100 items in each sample one would expect the rms error in the estimated mean probabilities to be about 1/√100=0.1 times the rms error in the 100 items in the sample so the mean probability estimates should be accurate to about 3 digits.

Correcting Faulty Observations

  One can use the results of two independent quality tests to improve the estimate of the probability of a finding a good item. The counts for Tester 1 of good and bad items are sesignated N₁ and N₂ and those of Tester 2 are N₃ and N₄ and their probabilities are p and q and p' and q' respectively. Based on the two observers' assessments of the items how can we determine N_G, N_B, p_G and p_B? The answer is to keep track of the number of times, N₁₃, when both N₁ and N₃ are good and the number of times, N₂₄, when both N₂ and N₄ are bad.

Then we can borrow a trick used by Rutherford to improve on the scintillation rates determined by two different observers.

Even with relatively large rates for the counting errors one still get a good estimate of the actual rates.

The mean values indicated were found by averaging the counts for N_G and N_B using N_G+N_B=N₀.

Supplemental (Sep 26): The estimated probabilities for G and B are again the averages for the 10 sets of assessments each involving 100 items. The set of stochastic variables used to generate the data were for p_G, a, d, a' and d' and were randomly set to 1 or 0 based on the rates as was done previously.

Supplemental (Sep 26): See Feller, An Introduction to Probability Theory and Its Applications, Vol I, 2nd ed., p. 160, prob. 23 which cites Rutherford. See also Rutherford &al., Probability Variations in the Distribution of α Particles cited in Rutherford's book linked above.

Uncertainty in the Cross Terms for the Comparison of the Two Quality Assessments

  When comparing the two processes for determining quality of items using the more accurate estimates good (G) and bad (B) and the less accurate estimates pass (P) and fail (F) the uncertainty  in the cross terms can be quite large. The conditional probabilities were determined as follows.

We can recalculate the Excel worksheet that generated the 10 sets of 100 random estimates of the quality of G, B, P, and F items to get a new set of averages and save the numerical values. With 100 of these trials the average of b and c were determined and as well as the root mean square deviations from this average.

One can see that there is quite a bit of variation in the cross terms even though the average turns out to be fairly accurate. A large number of tests of a given set of items is needed to get good estimates of the cross terms b and c in order to check theoretical results.

Faulty Sorts

  Consider a faulty sort mechanism intended to separate good from bad. If the decision depends on the perception of what's good and what's bad the sort can end up getting more mixed up instead of improving the sort. Using counts instead of probabilities one can show that the "sort" has an equilibrium distribution.

If A is the matrix containing the components a, b, c, d in the last blog then the limit after repeatedly multiplying an initial distribution by A can be determined with N₁=N_∞ and N₂=N₀-N_∞.

So if A represents a faulty processing mechanism used to sort successive distributions then the limit of the distributions will be (90, 10)^T. One can treat the column vectors of A as "distributions" with their limits being A^∞ above. One could use the equilibrium distribution to characterize the sort mechanism.

One can view a decision mechanism as a kind of sort. From this perspective it might be better to do an objective test of each item rather than rely some preconceived notions.

Note that the final result of the repeated sort is dependent on the process itself and not on whether or not a particular item is good or bad. It can degrade better sorts and improve poorer sorts.

Faulty Decisions

  Perhaps this might be a good time to review Bayesian inference and its application to decision making. Suppose we have two methods to assess the quality of items in a given sample. The first method is more exact but is also more difficult than the second. Let's call the results of the first test good (G) and bad (B) and those of the second pass (P) and fail (F). How might we go about comparing them?

We could assume that the probabilities of the two sets of outcomes are linearly related by matrix transforming the first set of probabilities to the second. Since the results are either G or B in the first case and P or F in the second only two components of the matrix are independent so we choose the variables to be the cross terms.

So given p, q, b and c one one can determine p' and q'. Alternatively one can take a statistical approach to the problem. In the example below ten sets of trials with N₀=100 were averaged to get estimates of the values for N₁, N₂, N₃, N₄, N₁₃, etc. Three stochastic variables s_p, s_a, s_d equal to 1 or 0 based on whether or not a random number between 0 and 1 inclusive was less than or equal to the probability associated with the variable were used.

The following results were obtained.

The conditional probabilities are p(i|j) or "the probability of outcome i given j" where i and j are the number of the cells or alternatively G, B, P and F. It was assumed that there was a 10% chance that a G item would test as F and a 20% chance that a B item would test as P. There was only a 1% chance that a passed item would be bad but a 27% chance that a failed item would be bad.

The probability of passing a bad item ends up about twice the probability of failing a good item but the chance of encountering a bad item is relatively rare.

Supplemental (Sep 23): More on conditional probability can be found in Parzen, Modern Probability Theory and Its Applications. The paper cited is Bayes, An Essay towards Solving a Problem in the Doctrine of Chances.

Supplemental (Sep 23): The values above for p' and q' are the estimated values. The calculated values are p'=0.865 and q'=0.135.