Modelling competing legal arguments using Bayesian networks

We have previously always tried to capture all of the competing hypotheses and evidence in a legal case in a single coherent Bayesian network model. But our new paper explains why this may not always be sensible and how to deal with it by using "competing" models. The full published version can be read here.

This work arose out of the highly successful Isaac Newton Institute Cambridge Programme on Probability and Statistics in Forensic Science.

Full reference:

Neil, M., Fenton, N. E., Lagnado, D. A. & Gill, R. (2019), "Modelling competing legal arguments using Bayesian Model Comparison and Averaging". Artificial Intelligence and Law https://doi.org/10.1007/s10506-019-09250-3 .The full published version can be read here

See also

• The Simonshaven murder case modelled as a Bayesian network
• Bayesian networks and argumentation 
• Challenging claims that probability theory is incompatible with legal reasoning
• Probability and statistics in forensics

Hannah Fry’s “Hello World” and the Example of Algorithm Bias

“Hello World” is an excellent book by Hannah Fry that provides lay explanations about both the potential and threats of AI and machine learning algorithms in the modern world. It is filled with many excellent examples, and one that is especially important is in Chapter 3 (“Justice”) about the use of algorithms in the criminal justice system. The example demonstrates the extremely important point that there is an inevitable trade-off between ‘accuracy’ and ‘fairness’ when it comes to algorithms that make decisions about people.

While the overall thrust and conclusions of the example are correct the need to keep any detailed maths out of the book might leave careful readers unconvinced about whether the example really demonstrates the stated conclusions. I feel it is important to get the details right because the issue of algorithmic fairness is of increasing importance for the future of AI, yet is widely misunderstood.

I have therefore produced a short report that provides a fully worked explanation of the example. I explain what is missing from Hannah's presentation, namely any explicit calculation of the false positive rates of the algorithm. I show how Bayes theorem (and some other assumptions) are needed to compute the false positive rates for men and women. I also show why and how a causal model of the problem (namely a Bayesian network model) makes everything much clearer.

Fry, H. (2018). "Hello world : how to be human in the age of the machine". New York: W. W. Norton & Company, Inc. 

My report:

 Fenton, N E. (2019)  "Hannah Fry’s 'Hello World' and the Example of Algorithm Bias", DOI 10.13140/RG.2.2.14339.55844

A pdf of the report is also available here

See also:

• Bias in AI algorithms 
• ProPublica Is Wrong In Charging Racial Bias In An Algorithm 

The Simonshaven murder case modelled as a Bayesian network

A paper published today in Topics in Cognitive Science is one in a series of analyses of a Dutch murder case, each using a different modelling approach. In this case a woman was murdered while out walking with her husband in a quiet recreational area near the village of Simonshaven, close to Rotterdam, in 2011. The trial court of Rotterdam convicted the victim’s husband of murder by intentionally hitting and/or kicking her in the head and strangling her. For the appeal the defence provided new evidence about other ‘similar’ murders in the area committed by a different person.

The idea to use this case to evaluate a number of different methods for modelling complex legal cases was originally proposed by Floris Bex (Utrecht), Anne Ruth Mackor (Groningen) and Henry Prakken (Utrecht). In September 2016 -as part of our Programme Probability and Statistics in Forensic Science at the Isaac Newton Institute Cambridge - a special two-day workshop was arranged in which different teams were presented with the Simonshaven evidence and had to produce a model analysis. At the time the Appeal was still to be heard. In a follow-up workshop to review the various solutions (held in London in June 2017 as part of the BAYES-KNOWLEDGE project) the participants agreed to publish their results in a special issue of a journal.

This paper describes the Bayesian Network (BN) team's solution. One of the key aims was to  determine if a useful BN could be quickly constructed using the previously established idioms-based approach (this provides a generic method for translating legal cases into BNs). The BN model described was built by the authors during the course of the workshop. The total effort involved was approximately 26 hours (i.e. an average of 6 hours per author). With the basic assumptions described in the paper, the posterior probability of guilt once all the evidence is entered is 74%. The paper describes a formal evaluation of the model, using sensitivity analysis, to determine how robust the model conclusions are to key subjective prior probabilities over a full range of what may be deemed ‘reasonable’ from both defence and prosecution perspectives. The results show that the model is reasonably robust - pointing generally to a reasonably high posterior probability of guilt, but also generally below the 95% threshold expected in criminal law.

The authors acknowledge the insights of  the following workshop participants: Floris Bex, Christian Dahlman, Richard Gill, Anne Ruth Mackor, Ronald Meester, Henry Prakken, Leila Schneps, Marjan Sjerps, Nadine Smit, Bart Verheij, and Jacob de Zoete.


Full reference:

Fenton, N. E., Neil, M., Yet, B., & Lagnado, D. A. (2019). "Analyzing the Simonshaven Case using Bayesian Networks". Topics in Cognitive Science, 10.1111/tops.12417.  For those without a subscription to the journal, the published version can be read here: https://rdcu.be/bqYxp)  

See also:

• Bayesian networks and argumentation 
• Challenging claims that probability theory is incompatible with legal reasoning
• Probability and statistics in forensics

Challenging claims that probability theory is incompatible with legal reasoning

 
A new paper published in Science and Justice exposes why common claims that probability theory is incompatible with the law are flawed.

One of the most effective tactics that has been used by legal scholars to 'demonstrate' the 'limitations' and 'incompatibility' of probability theory (and particularly Bayes theorem) with legal reasoning is the use of puzzles like the following:

Fred is charged with a crime. A reliable eye witness testifies that someone exactly matching Fred’s appearance was seen fleeing the crime scene. But Fred is known to have an identical twin brother. So is the evidence relevant?"**

The argument to suggest that this example demonstrates probability theory is incompatible with legal norms goes something like this:

Both intuitively and legally it is clear that the evidence should be considered relevant. But according to probability theory (Bayes' theorem), the evidence has 'no probative value' since it provides no change in our belief about whether Fred is more likely than his twin brother to have been at the crime scene. Hence, according to probability theory the evidence is wrongly considered inadmissible.

Specifically, such problems are intended to show that use of probability theory results in legal paradoxes. As such, these problems have been a powerful detriment to the use of probability theory  in the law.

The new paper  shows that all of these puzzles only lead to ‘paradoxes’ under an artificially constrained view of probability theory and the use of the so-called likelihood ratio, in which multiple related hypotheses and pieces of evidence are squeezed into a single hypothesis variable and a single evidence variable. When the distinct relevant hypotheses and evidence are described properly in a causal model (a Bayesian network), the paradoxes vanish. Moreover, the resulting Bayesian networks provide a powerful framework for legal reasoning.

Full reference details of the paper:

de Zoete, J., Fenton, N. E., Noguchi, T., & Lagnado, D. A. (2019). "Countering the ‘probabilistic paradoxes in legal reasoning’ with Bayesian networks". Science & Justice 10.1016/j.scijus.2019.03.003. 

The pre-publication version (pdf)

The models (which can be run using AgenaRisk)

Two other papers just accepted (details to follow) also demonstrate the power of Bayesian networks in legal reasoning:

Fenton, N. E., Neil, M., Yet, B., & Lagnado, D. A. (2019). "Analyzing the Simonshaven Case using Bayesian Networks". Topics in Cognitive Science, 10.1111/tops.12417.  (Update: this had now been published; the published version can be read https://rdcu.be/bqYxp )

Neil, M., Fenton, N. E., Lagnado, D. A. & Gill, R. (2019), "Modelling competing legal arguments using Bayesian Model Comparison and Averaging". to appear Artififical Intelligence and Law . The full published version can be read here.

**This particular puzzle is easy to 'resolve'. The 'non-probative' Bayes conclusion is only correct if we assume that the only people who could possibly have committed the crime are Fred and his twin brother. In practice we have to consider the possibility that neither committed the crime. While the eye witness evidence fails to distinguish between which of Fred and his twin was at the crime scene the evidence results in the probability that Fred was at the crime scene increasing in relation to the hypothesis that Fred was not at the crime scene. 

Bayesian networks for critical maintenance decisions on the railway network

An important recent paper (published in the Journal of Risk and Reliability) by Haoyuan Zhang and William Marsh of Queen Mary University of London presents a Bayesian network model that can be used for maintenance decision support that is especially relevant for rail safety. The model overcomes the practical limitations of previous statistical models that have attempted to maximise asset reliability cost-effectively, by scheduling maintenance based on the likely deterioration of an asset. The model extends an existing statistical model of asset deterioration, but shows how

1. data on the condition of assets available from their periodic inspection can be used 
2. failure data from related groups of asset can be combined using judgement from experts 
3. expert knowledge of the causes of deterioration can be combined with statistical data to adjust predictions. 

The model (which was developed using the AgenaRisk software) is applied to a case study of bridges on the rail network in the UK.

A full pre-publication version is available here.

The full publication details for the paper are:

Zhang, H., & R Marsh, D. W. (2018). "Generic Bayesian network models for making maintenance decisions from available data and expert knowledge". Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 232(5), 505–523. https://doi.org/10.1177/1748006X17742765