## Thursday, 4 February 2016

### Problems with the Likelihood Ratio method for determining probative value of evidence: the need for exhaustive hypotheses

Norman Fenton, 4 Feb 2016

I have written several times before about the likelihood ratio (LR) method that is recommended for use by forensic scientists when presenting evidence (such as the fact that DNA collected at a crime scene is found to have a profile that matches the DNA profile of a defendant in a case). In general the LR is a very good and simple method for communicating the impact of evidence (in this case on the hypothesis that the defendant was at the crime scene), but its correct use is based on strict assumptions that have been routinely ignored by forensic experts and statisticians, leading to the very kind of confusion and misunderstanding (when presented to lawyers and juries) that it was supposed to help avoid. The papers [1] and [2] provide an in-depth analysis of the problems. In this short article I will highlight just one of these problems which invalidate the LR. Subsequent articles will focus on the other problems and issues.

To recap: The LR is the probability of finding the evidence E if the prosecution hypothesis Hp is true (formally we write this as 'Probability of E given Hp') divided by the probability of finding the evidence E if the defence hypothesis Hd is true (formally we write this as  'probability of E given Hd').

So, to compute the LR, the forensic expert is forced to consider the probability of finding the evidence under both the prosecution and defence hypotheses.  This is a very good thing to do because it helps to avoid common errors of communication that can mislead lawyers and juries (notably the prosecutor's fallacy). Even more importantly, the LR is a measure of the probative value of the evidence because:
• when the LR is greater than one the evidence supports the prosecution hypothesis (increasingly for larger values);
• when the LR is less than one it supports the defence hypothesis (increasingly as the LR gets closer to zero);
• when the LR is equal to one then the evidence supports neither hypothesis and so is 'neutral'. In such cases, since the evidence has no probative value lawyers and forensic experts believe it should not be admissible.
However, as explained in [1] and [2] (because of Bayes Theorem) for the LR to 'work' with respect to being a measure of probative value, the two hypotheses considered must be 'mutually exclusive and exhaustive'. This means that the defence hypothesis Hd must simply be the negation of the prosecution hypothesis Hp. So, for example, if Hp is "Defendant was at the crime scene" then Hp must be "Defendant was not at the crime scene".  Now, while there is more or less unanimity within the statistics and forensics field that the hypotheses must be mutually exclusive in order for the LR to be used, there is no such unanimity about the hypotheses being exhaustive. Indeed, the Royal Statistical Society Practitioner Guide to Case Assessment and Interpretation of Expert Evidence Guidelines [3] (page 32) specifies that the LR requires two mutually exclusive but not necessarily exhaustive hypotheses (which, interestingly, contradicts what is stated in the earlier Guidelines by the same group [4], page 96). To see why incorrect conclusions may be drawn when the hypotheses are not exhaustive we consider a very simple example:

Fred is the defendant for a crime.  The main evidence against Joe is that his DNA profile is found to be a match of a DNA sample found at the scene of the crime (for simplicity we ignore the possibility of errors in the DNA match). The DNA profile is of a type that is found in only 1 in 10,000 people. However, Fred has an identical twin brother Joe. Using the following:
• Prosecution hypothesis Hp:  "Fred is the source of the DNA"
• Defence hypothesis Hd: "Joe is the source of the DNA"
and
• Evidence E: "the DNA found matches Fred's profile"
The defence reasons - correctly using the likelihood ratio approach- that the evidence E has no probative value with respect to the above two hypotheses, because the twins have the same DNA profile, i.e.
P(E given Hp) = P(E given Hd) = 1.
Hence, the defence demands the evidence is withdrawn because it is 'neutral'.

The problem here is that, even if we assume the hypotheses are mutually exclusive (i.e. we exclude the possibility that both the twins committed the crime) they are certainly NOT exhaustive. The correct defence hypothesis in this case should be "Fred is NOT the source of the DNA". This is made up of two cases:
• Hd: "Joe is the source of the DNA"
• Ho: "Another person (not Fred or  Joe) is the source of the DNA"
If we assume - before any evidence is known - that Hp, Hd and Ho are equally likely then the impact of observing the evidence is certainly NOT neutral - it is probative in favour of the prosecution hypothesis as can be shown from running the calculations in a Bayesian network tool:

The probability of Hp increases from 33% to to just under 50%.

But the supposedly 'neutral' evidence can have an even more dramatic impact in practice. Suppose, for example, that Joe has an alibi that is considered pretty reliable. Then this might reduce our prior belief in his innocence to 2%. In this case the before and after probabilities are:

The belief in the prosecution hypothesis in this case has shifted to above 95% - possibly sufficient for a jury to be convinced it is the truth.

If the DNA evidence in the above example was a non-match then the LR approach using the original hypotheses is even more obviously flawed because in this case:
P(E given Hp) = P(E given Hd) = 0
But the evidence is certainly anything but 'neutral' because, after observing the evidence, the prosecution hypothesis Hp must be false (as must Hd).

While the example above is obviously simplistic and contrived more realistic examples are provided in [1] which also highlights this very problem in the case of Barry George (convicted and subsequently acquitted of the murder of TV celebrity Gill Dando after an appeal ruled that the gunpowder residue evidence presented in the original trial was inadmissible in a re-trial on the basis that it had a LR equal to one and so had 'no probative value'.)

References
1. Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, (2013). "When ‘neutral’ evidence still has probative value (with implications from the Barry George Case)", Science and Justice, http://dx.doi.org/10.1016/j.scijus.2013.07.002.  A pre-publication draft of the article can be found here.
2. Fenton N.E, Neil M, Berger D, “Bayes and the Law”, Annual Review of Statistics and Its Application, Volume 3, 2016 to appear. Pre-publication version here
3. Jackson, G., Aitken, C., & Roberts, P. (2015). PRACTITIONER GUIDE NO 4: Case Assessment and Interpretation of Expert Evidence. Royal Statistical Society.  Available here.
4. Aitken, C, Roberts, P, Jackson, G, (2010) PRACTITIONER GUIDE NO 1:"Fundamentals of Probability and Statistical Evidence in Criminal Proceedings: Guidance for Judges, Lawyers, Forensic Scientists and Expert Witnesses. Royal Statistical Society. Available here.

1. Check Out the Newest Cloud-Based Risk Management System at RIMS 2016 April 10th to 13th 2016 – San Diego, CA
Rims 2016

2. A HEART OVER INDEBTED AND FULL OF GRATITUDE TO DR OGUDU THE MIGHTIEST LOTTERY SPELL CASTER IN THE WORLD
My name is JOHN REHM It all started 2 years ago when i lost my job, and things became so hard for me and my family living in Florida and i decided to use my little savings to purchase a Florida lottery which never yield a result, I was now leaving in total mess and much debt i promised myself that i will never play the lottery again, After months of joblessness i was reading a tread online about{ DR OGUDU THE LOTTERY SPELL CASTER} and how he has made so many people to win the lottery, I for one don,t believe in magic and black powers but because i was very frustrated and stranded i decided to contact him through his email address, He demanded for few details about me which i gave him under 48 hours he gave me some numbers to play which have changed my life today.i was in a bar drinking when Florida Lottery announces that MR JOHN REHM is the sole winner of the 1,000,000 million POWER BALL jackpot from the drawing held on Wednesday, January 26, should claimed the prize at Lottery Headquarters in Tallahassee. if you,re in need of such winnings contact him via [OGUDUSPELLTEMPLE@YAHOO.COM] OR WHATS-APP HIM +2348106058254

A HEART OVER INDEBTED AND FULL OF GRATITUDE TO DR OGUDU THE MIGHTIEST LOTTERY SPELL CASTER IN THE WORLD
My name is JOHN REHM It all started 2 years ago when i lost my job, and things became so hard for me and my family living in Florida and i decided to use my little savings to purchase a Florida lottery which never yield a result, I was now leaving in total mess and much debt i promised myself that i will never play the lottery again, After months of joblessness i was reading a tread online about{ DR OGUDU THE LOTTERY SPELL CASTER} and how he has made so many people to win the lottery, I for one don,t believe in magic and black powers but because i was very frustrated and stranded i decided to contact him through his email address, He demanded for few details about me which i gave him under 48 hours he gave me some numbers to play which have changed my life today.i was in a bar drinking when Florida Lottery announces that MR JOHN REHM is the sole winner of the 1,000,000 million POWER BALL jackpot from the drawing held on Wednesday, January 26, should claimed the prize at Lottery Headquarters in Tallahassee. if you,re in need of such winnings contact him via [OGUDUSPELLTEMPLE@YAHOO.COM] OR WHATS-APP HIM +2348106058254

3. hello viewers this is certainly a shocking and a genuine live Testimony! It was hard for me to believe when i saw a comment on the internet regarding how Dr.eguaga help people winning lottery by sending them the winning numbers. This winning numbers are 100% guarantee that you must win the lottery. I give it a try and i won £2,000,000. My dreams came through when i email Dr eguaga and tell him i need the lottery numbers. i have come a long way spending money on ticket just to make sure i win. But i never know that winning was so easy until the day i meant the spell caster online which so many people has talked about that he is very great in casting lottery spell, so i decide to give it a try.I contacted this man and he did a perfect spell and he gave me the winning lottery numbers. I won (Two Millions Euro) Dr.Eguaga truly you are the best, with these man you can win millions of money through lottery. Email him:eguagavoodospellcaster@gmail.com. you can also whats app him on this number:+2347010511571

4. hello viewers this is certainly a shocking and a genuine live Testimony! It was hard for me to believe when i saw a comment on the internet regarding how Dr.eguaga help people winning lottery by sending them the winning numbers. This winning numbers are 100% guarantee that you must win the lottery. I give it a try and i won £2,000,000. My dreams came through when i email Dr eguaga and tell him i need the lottery numbers. i have come a long way spending money on ticket just to make sure i win. But i never know that winning was so easy until the day i meant the spell caster online which so many people has talked about that he is very great in casting lottery spell, so i decide to give it a try.I contacted this man and he did a perfect spell and he gave me the winning lottery numbers. I won (Two Millions Euro) Dr.Eguaga truly you are the best, with these man you can win millions of money through lottery. Email him:eguagavoodospellcaster@gmail.com. you can also whats app him on this number:+2347010511571

5. hello viewers this is certainly a shocking and a genuine live Testimony! It was hard for me to believe when i saw a comment on the internet regarding how Dr.eguaga help people winning lottery by sending them the winning numbers. This winning numbers are 100% guarantee that you must win the lottery. I give it a try and i won £2,000,000. My dreams came through when i email Dr eguaga and tell him i need the lottery numbers. i have come a long way spending money on ticket just to make sure i win. But i never know that winning was so easy until the day i meant the spell caster online which so many people has talked about that he is very great in casting lottery spell, so i decide to give it a try.I contacted this man and he did a perfect spell and he gave me the winning lottery numbers. I won (Two Millions Euro) Dr.Eguaga truly you are the best, with these man you can win millions of money through lottery. Email him:eguagavoodospellcaster@gmail.com. you can also whats app him on this number:+2347010511571