The likelihood ratio and why its use in forensic analysis is often flawed

FORREST 2016 (for details see here)

I am giving the opening address at the Forensic Institute 2016 Conference (FORREST 2016) in Glasgow on 5 July 2016. The talk is about the benefits and pitfalls of using the likelihood ratio to help understand the impact of forensic evidence. The powerpoint slide show for my talk is here.

While a lot of the material is based on our recent Bayes and the Law paper, there is a new simple example of the danger of using the likelihood ratio (LR) when the defence hypothesis is not the negation of the prosecution hypothesis. Recall that the LR for some evidence E is the probability of E given the prosecution hypothesis divided by the probability of E given the defence hypothesis. The reason the LR is popular is because it is a measure of the probative value of the evidence E in the sense that:

• LR>1 means E supports the prosecution hypothesis
• LR<1 means  E supports the defence hypothesis
• LR=1 means E has no probative value

This follows from Bayes Theorem but only when the defence hypothesis is the negation of the prosecution hypothesis. The problem is that there are Forensic Science Guidelines* that explicitly state that this requirement is not necessary. But if the requirement is not met then it is possible to have LR<1 even though E actually supports the prosecution hypothesis. Here is the example:

A raffle has 100 tickets numbered 1 to 100

Joe buys 2 tickets and gets numbers 3 and 99

The ticket is drawn but is blown away in the wind.

Joe says he had the winning ticket, but the organisers say neither 3 nor 99 was not the winning ticket. In this case the prosecution hypothesis H is “Joe won the raffle” (i.e. either 3 or 99 was the winning ticket).

Suppose we have the following evidence E presented by a totally reliable eye witness:

 

E: “winning ticket was an odd nineties number (i.e. 91, 93, 95, 97, or 99)”

Does the evidence E support H? let's do the calculations:

• Probability of E given H = 1/2
• Probability of E given not H = 4/98

So the LR  is  (1/2)/(4/98) = 12.25

That means the evidence CLEARLY supports H. In fact, the probability of H increases from a prior of 1/50 to a posterior of 1/5, so there is no doubt it is supportive.

But suppose the organisers’ assert that their (defence) hypothesis is:

H’: “Winning ticket was a number between 95 and 97”

Then in this case we have:

• Probability of E given H = ½
• Probability of E given H’ = 2/3

So the LR  is  ( 1/2)/(2/3) = 0.75

That means that in this case the evidence supports H’ over H. The problem is that, while the LR does indeed 'prove' that the evidence is more supportive of H' than H that is actually irrelevant unless there is other evidence that proves that H' is the only possible alternative to H (i.e. that H' equivalent to 'not H').  In fact, the  'defence' hypothesis has been cherry picked. The evidence E supports H irrespective of which cherry-picked alternative is considered.  

Norman Fenton, 1 July 2016
 

*Jackson G, Aitken C, Roberts P. 2013. Practitioner guide no. 4. Case assessment and interpretation of expert evidence: guidance for judges, lawyers, forensic scientists and expert witnesses. London: R. Stat. Soc. http://www.maths.ed.ac.uk/∼cgga/Guide-4-WEB.pdf.  Page 29: "The LR is the ratio of two probabilities, conditioned on mutually exclusive (but not necessarily exhaustive) propositions."

See also:

• Problems with the Likelihood Ratio method for determining probative value of evidence: the need for exhaustive hypotheses
• Misleading DNA evidence
• Barry George case: new insights on the evidence