The Cab accident problem: new insights into probabilistic reasoning when there is uncertainty about the witness reliability

 

One of the classic problems used to evaluate how well lay people perform probabilistic updating is the "Blue/Green Cab accident problem" (or equivalently with buses). The problem is usually expressed as follows:

 A cab was involved in a hit-and-run accident at night. Two cab companies, the Green and the Blue, operate in the city. 

90% of the cabs in the city are Green and 10% are Blue. 

A witness identified the cab as Blue. The court tested the reliability of the witness under the circumstances that existed on the night of the accident and concluded that the witness correctly identified each of the two colours 80% of the time and failed 20% of the time. 

What is the probability that the cab involved in the accident was Blue rather than Green? 

A major finding of the original study was that many participants neglected the population base rate data (i.e. that 90% of the cabs are Green) entirely in their final estimate, simply giving the witness’s accuracy (80%) as their answer. 

However, what was not considered in this and similar studies was any uncertainty about the witness reliability (this is called second order uncertainty). If the 80% figure was based on 80 correct answers in 100 then the 80% estimate seems reasonable. But what if it was based only on 5 tests in which the witness was correct in 4? In that case there is much more uncertainty about the "80%" figure; using a Bayesian network model to get the correct 'nomative' solution, it can be shown that while the witness's report does increase the probability of the cab being Blue, it simultaneously decreases our estimate of their future accuracy (because Blue cabs are so uncommon).

A new paper by lead author Stephen Dewitt that addresses how well lay people reason about this second order uncertainty has been published in Frontiers in Psychology. It was based on a study of 131 participants,  who were asked to update their estimates of both the probability the cab involved was Blue, as well as the witness's accuracy, after they claim it was Blue. While some participants responded normatively, most wrongly assumed that one of the probabilities was a certainty; for example, a quarter assumed the cab was Green, and thus the witness was wrong, decreasing their estimate of their accuracy. Half of the participants refused to make any change to the witness reliability estimate. 

Dewitt, S., Fenton, N. E., Liefgreen, A,  & Lagnado, D. A. (2020). "Propensities and second order uncertainty: a modified taxi cab problem". Frontiers in Psychology, https://doi.org/10.3389/fpsyg.2020.503233