Monday, 24 December 2018

Bayesian network approach to Drug Economics Decision Making


This is an update of a short paper I first produced in 2014.

Consider the following problem:
A relatively cheap drug (drug A) has been used for many years to treat patients with disease X. The drug is considered quite successful since data reveals that 85% of patients using it have a ‘good outcome’ which means they survive for at least 2 years. The drug is cheap and the overall “financial benefit” of the drug (which assumes a ‘good outcome’ is worth $5000 and is defined as this figure minus the cost) has a mean of $4985.

There is an alternative drug (drug B) that a number of specialists in disease X strongly recommend. However, the data reveals that only 65% of patients using drug B survive for at least 2 years. Moreover, this drug is expensive. The overall “financial benefit” of the drug has a mean of just $2777.
On seeing the data the Health Authority recommends a ban against the use of drug B. Is this a rational decision?

The answer turns out to be no. This short paper explains this using a simple Bayesian network model that you can run (by downloading the free copy of AgenaRisk). Moreover, you can also compute the optimal decision automatically using the Hybrid Influence Diagram tool in AgenaRisk.


Fenton N.E. (2018) "A Bayesian Network and Influence Diagram for a simple example of Drug Economics Decision Making",  DOI: https://doi.org/10.13140/RG.2.2.33659.77600

Thursday, 20 December 2018

Review of “The Book of Why" by Pearl and Mackenzie

Judea Pearl and Dana Mackenzie: “The Book of Why: The New Science of Cause and Effect”, Basic Books, 2018. ISBN: 9780465097609

 www.basicbooks.com/titles/judea-pearl/the-book-of-why/9780465097609/
We have finally completed a detailed review of this important and outstanding book - the review will hopefully be published in the journal Artificial Intelligence. But a preprint of the full review is now available.

Some excerpts from the review:
  • Judea Pearl, a Turing Award prize winner, is a true giant of the field of computer science and artificial intelligence. The Turing award is the highest distinction in computer science; i.e., the Nobel Prize of computing. To say that his new book with Dana Mackenzie is timely is, in our view, an understatement. Coming from somebody of his stature and being written for a general audience (unlike his previous books), means that the concerns we have held about both the limitations of solely data driven approaches to artificial intelligence (AI) and the need for a causal approach, will finally reach a very broad audience.
  • According to Pearl, the state of the art in AI today is merely a ‘souped-up’ version of what machines could already do a generation ago: find hidden regularities in a large set of data. “All the impressive achievements of deep learning amount to just curve fitting”, he said recently. 
  • In Chapter 1, the core message about the need for causal models is underpinned by what Pearl calls “The Ladder of Causation”, which is then used to orient the ideas presented throughout the book. Pearl’s ladder of causation suggests that there are three steps to achieving true AI. .... Pearl also characterises these three steps on the ladder as 1) ‘seeing’; 2) ‘doing’; and 3) ‘imagining’. 
  • One of the reasons ‘deep learning’ has been so successful is that many problems can be solved by optimisation alone without the need to even consider advancing to rungs in the ladder of causation beyond the first. These problems include machine vision and machine listening, natural language processing, robot navigation, as well as other problems that fall within the areas of clustering, pattern recognition and anomaly detection. Big data in these cases is clearly very important and the advances being made using deep learning are undoubtedly impressive, but Pearl convincingly argues that they are not AI.
  • There is much excellent material in this book but, for us, the two key messages are: 1) “True AI” cannot be achieved by data and curve fitting alone, since causal representation of the underlying problems is also required to answer “what-if” questions, and 2) Randomized control trials are not the only ‘valid’ method for determining causal effects.
Norman Fenton, Martin Neil, and Anthony Constantinou, 20 December, 2018

For the full review see:
Review of: Judea Pearl and Dana Mackenzie: “The Book of Why: The New Science of Cause and Effect”, Basic Books, 2018 DOI: https://doi.org/10.13140/RG.2.2.27512.49925, by Norman Fenton, Martin Neil, and Anthony Constantinou

Thursday, 29 November 2018

AI for healthcare requires ‘smart data’ rather than ‘big data’

Norman Fenton gave a talk titled AI for healthcare requires ‘smart data’ rather than ‘big data’ to medics at the Royal London Hospital on 27 November. He explained the background and context for the PAMBAYESIAN project.

Norman's Powerpoint presentation

Thursday, 15 November 2018

Book Launch at the Turing Institute

Some photos from last night's book launch event at The Turing Institute

Norman Fenton and Martin Neil

 

More photos

Tuesday, 13 November 2018

Book Launch event at The Turing Institute

On 14 November 2018 Norman Fenton and Martin Neil are hosting a reception at The Turing Institute to celebrate the launch of the Second Edition of their book "Risk Assessment and Decision Analysis with Bayesian Networks".


Slide show of the book 

A small number of places remain for people to register for the reception

Book Blog

Tuesday, 30 October 2018

Forensic Science in Criminal Justice


Norman Fenton was a contributor to The Turing Institute submission to the House of Lords Science and Technology Committee inquiry into Forensic Science, published on the Parliament website.

Queen Mary News has a nice article summarising the contributions of its staff members.

The Turing Institute summary and full report.






Wednesday, 17 October 2018

Turing Fellows

Proud to announce that five members of the Risk and Information Management Group have been appointed as Fellows of the Turing Institute.  They are: Anthony Constantinou, Norman Fenton, William Marsh, Martin Neil, and Fabrizio Smeraldi. 


The official Turing Institute announcement is here.


Wednesday, 26 September 2018

Bayesian networks for trauma prognosis


There is an excellent online resource produced by Barbaros Yet that summarises the results of collaboration between the Risk and Information Management research group at Queen Mary and the Trauma Sciences Unit, Barts and the London School of Medicine and Dentistry. This work focused on developing Bayesian network (BN) models to improve decision support for trauma patients.

The website not only describes two BN models in detail (one for predicting acute traumatic coagulopathy in early stage of trauma care and one for predicting the outcomes of traumatic lower extremities with vascular injuries) but allows you to run the models in real time showing summary risk calculations after you enter observations about a patient.

The models are powered by AgenaRisk.

Links:

  • http://traumamodels.com/
  •  Perkins ZB, Yet B, Glasgow S, Marsh DWR, Tai NRM, Rasmussen TE (2018). “Long-term, patient centered outcomes of Lower Extremity Vascular Trauma”, Journal of Trauma and Acute Surgery. DOI:10.1097/TA.0000000000001956 
  • Yet B, Perkins ZB, Tai NR, and Marsh DWR (2017). “Clinical Evidence Framework for Bayesian Networks” Knowledge and Information Systems, 50(1), pp.117-143.DOI:10.1007/s10115-016-0932-1 
  •  Perkins ZB, Yet B, Glasgow S, Cole E, Marsh W, Brohi K, Rasmussen TE, Tai NRM (2015). “Meta-analysis of prognostic factors for amputation following surgical repair of lower extremity vascular trauma” British Journal of Surgery, 12 (5), pp. 436-450. DOI:10.1002/bjs.9689
  • Yet B, Perkins ZB, Rasmussen TE et al.(2014). Combining data and meta-analysis to build Bayesian networks for clinical decision support. J Biomed Inform vol. 52, 373-385. http://dx.doi.org/10.1016/j.jbi.2014.07.018   http://qmro.qmul.ac.uk/xmlui/handle/123456789/23055
  • Perkins ZB, Yet B, Glasgow S, Cole E, Marsh W, Brohi K, Rasmussen TE, Tai NRM (2015). “Meta-analysis of prognostic factors for amputation following surgical repair of lower extremity vascular trauma” British Journal of Surgery, 12 (5), pp. 436-450. DOI:10.1002/bjs.9689
  • Yet B, Perkins ZB, Rasmussen TE, Tai NR, and Marsh DWR (2014). “Combining Data and Meta-analysis to Build Bayesian Networks for Clinical Decision Support” Journal of Biomedical Informatics , 52, pp.373-385. DOI:10.1016/j.jbi.2014.07.018
  • Yet B, Perkins Z, Fenton N et al.(2014). Not just data: a method for improving prediction with knowledge. J Biomed Inform vol. 48, 28-37. http://dx.doi.org/10.1016/j.jbi.2013.10.012
  • Yet B, Perkins Z, Tai N et al.(2014). Explicit evidence for prognostic Bayesian network models. Stud Health Technol Inform vol. 205, 53-57. http://dx.doi.org/10.3233/978-1-61499-432-9-53
  • Perkins Z, Yet B, Glasgow S et al. (2013). EARLY PREDICTION OF TRAUMATIC COAGULOPATHY USING ADMISSION CLINICAL VARIABLES. SHOCK. vol. 40, 25-25.

Friday, 24 August 2018

Second Edition of our book to be published 28 August 2018




From the back cover of the Second Edition:

************************************************
"The single most important book on Bayesian methods for decision analysts" —Doug Hubbard (author in decision sciences and actuarial science)  
"The book provides sufficient motivation and examples (as well as the mathematics and probability where needed from scratch) to enable readers to understand the core principles and power of Bayesian networks." —Judea Pearl (Turing award winner)  
"The lovely thing about Risk Assessment and Decision Analysis with Bayesian Networks is that it holds your hand while it guides you through this maze of statistical fallacies, p-values, randomness and subjectivity, eventually explaining how Bayesian networks work and how they can help to avoid mistakes.” —Angela Saini (award-winning science journalist, author & broadcaster)
Since the first edition of this book published, Bayesian networks have become even more important for applications in a vast array of fields. This second edition includes new material on influence diagrams, learning from data, value of information, cybersecurity, debunking bad statistics, and much more. Focusing on practical real-world problem-solving and model building, as opposed to algorithms and theory, it explains how to incorporate knowledge with data to develop and use (Bayesian) causal models of risk that provide more powerful insights and better decision making than is possible from purely data-driven solutions.

Features
  • Provides all tools necessary to build and run realistic Bayesian network models
  • Supplies extensive example models based on real risk assessment problems in a wide range of application domains provided; for example, finance, safety, systems reliability, law, forensics, cybersecurity and more
  • Introduces all necessary mathematics, probability, and statistics as needed
  • Establishes the basics of probability, risk, and building and using Bayesian network models, before going into the detailed applications
A dedicated website contains exercises and worked solutions for all chapters along with numerous other resources. The AgenaRisk software contains a model library with executable versions of all of the models in the book. Lecture slides are freely available to accredited academic teachers adopting the book on their course.

************************************************

Sample chapters are available on the book's website

Thursday, 26 July 2018

Updating Prior Beliefs Based on Ambiguous Evidence


Suppose two nations, North Bayesland and South Bayesland are independently testing new missile technology. Each has made six detonation attempts: North Bayesland has been successful once and  South Bayesland four times. You observe another detonation on the border between the two countries but cannot determine the source. Based only on the provided information:
  1. What is the probability that North (or South) Bayesland is the source of this missile? 
  2. What is your best estimate of the propensity for success of North and South Bayesland after this latest observation (i.e. the probability, for each nation, that a future missile they launch will detonate)?
The general form of this problem is ubiquitous in many areas of life.  But how well do people answer such questions?

Our paper "Updating Prior Beliefs Based on Ambiguous Evidence", which was accepted at the prestigious 40th Annual Meeting of the Cognitive Science Society (CogSci 2018) in Madison, Wisconsin, addresses this problem. Stephen Dewitt (former QMUL PhD student) is presenting the paper on 27 July. 

First of all the normative answer to Question 1 - based on simple Bayesian reasoning - is 20% for North Bayesland and 80% for South Bayesland. But Question 2 is much more complex because we cannot assume the small amount of data on previous detonation attempts represents a 'fixed' propensity of success (the normative Bayesian solution requires a non-trivial Bayesian network that models our uncertainty about the success propensities).

Based on experiments involving 250 paid participants, we discovered two types of errors in the answers.
  1. There was a ‘double updating’ error: individuals appear to first use their prior beliefs to interpret the evidence, then use the interpreted form of the evidence, rather than the raw form, when updating. 
  2. We found an error where individuals convert from a probabilistic representation of the evidence to a categorical one and use this representation when updating. 
Both errors have the effect of exaggerating the evidence in favour of the solver’s prior belief and could lead to confirmation bias and polarisation. Given the importance of the class of problems to which the study applies, we believe that greater understanding of the cognitive processes underlying the errors should therefore be an important avenue for future study.

The full paper details and pdf (also available here)
Dewitt, S, Lagnado, D, Fenton N. E (2018), "Updating Prior Beliefs Based on Ambiguous Evidence", CogSci 2018, Madison Wisconsin, 25-28 July 2018 
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), under Contract [2017-16122000003]. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. Funding was also provided by the ERC project ERC-2013-AdG339182-BAYES_KNOWLEDGE and the Leverhulme Trust project RPG-2016-118 CAUSAL-DYNAMICS.

UPDATE: Stephen Dewitt presenting the paper in Madison:


Saturday, 14 July 2018

How to handle uncertain priors in Bayesian reasoning

In the classic simple Bayesian problem we have:
  • a hypothesis H (such as 'person has specific disease') with a prior probability (say 1 in a 1000) and
  • evidence E (such as a test result which may be positive or negative for the disease) for which we know the probability E given H (for example the probability of a false positive is 5% and the probability of a false negative is 0%). 
With those particular values Bayes' theorem tells us that a randomly selected person who tests positive has a 1.96% probability of having the disease.

But what if there is uncertainty about the prior probabilities (i.e. the 1 in a 1000, the 5% and 0%). Maybe the 5% means 'anywhere between 0 and 10%'. Maybe the 1 in a 1000 means we only saw it once in 1000 people. This new technical report explains how to properly incorporate uncertainty about the priors using a Bayesian Network.


Fenton NE, "Handling Uncertain Priors in Basic Bayesian Reasoning", July 2018,  DOI 10.13140/RG.2.2.16066.89280

Friday, 13 July 2018

How much do we trust academic 'experts'?


Queen Mary has released the following press release about our new paper: Osman, M., Fenton, N. E., Pilditch, T., Lagnado, D. A., & Neil. M. (2018). "Who do we trust on social policy interventions", to appear next week in the journal Basic and Applied Social Psychology (6/8/18 update: the published paper is here). The preprint of the paper is here. There are already a number of press reports on it (see below).

People trust scientific experts more than the government even when the evidence is outlandish


Members of the public in the UK and US have far greater trust in scientific experts than the government, according to a new study by Queen Mary University of London. In three large scale experiments, participants were asked to make several judgments about nudges -behavioural in interventions designed to improve decisions in our day-to-day lives.

The nudges were introduced either by a group of leading scientific experts or a government working group consisting of special interest groups and policy makers. Some of the nudges were real and had been implemented, such as using catchy pictures in stairwells to encourage people to take the stairs, while others were fictitious and actually implausible like stirring coffee anti-clockwise for two minutes to avoid any cancerous effects.

The study, published in Basic and Applied Social Psychology, found that trust was higher for scientists than the government working group, even when the scientists were proposing fictitious nudges. Professor Norman Fenton, from Queen Mary’s School of Electronic Engineering and Computer Science, said: “While people judged genuine nudges as more plausible than fictitious nudges, people trusted some fictitious nudges proposed by scientists as more plausible than genuine nudges proposed by government. For example, people were more likely trust the health benefits of coffee stirring than exercise if the former was recommended by scientists and the latter by government.”

The results also revealed that there was a slight tendency for the US sample to find the nudges more plausible and more ethical overall compared to the UK sample. Lead author Dr Magda Osman from Queen Mary’s School of Biological and Chemical Sciences, said: “In the context of debates regarding the loss of trust in experts, what we show is that in actual fact, when compared to a government working group, the public in the US and UK judge scientists very favourably, so much so that they show greater levels of trust even when the interventions that are being proposed are implausible and most likely ineffective. This means that the public still have a high degree of trust in experts, in particular, in this case, social scientists.” She added: “The evidence suggests that trust in scientists is high, but that the public are sceptical about nudges in which they might be manipulated without them knowing. They consider these as less ethical and trust the experts proposing them less with nudges in which they do have an idea of what is going on.”

Nudges have become highly popular decision-support methods used by governments to help in a wide range of areas such as health, personal finances, and general wellbeing. The scientific claim is that to help people make better decisions regarding their lifestyle choices, and those that improve the welfare of the state, it is potentially effective to subtly change the framing of the decision-making context, which makes the option which maximises long term future gains more prominent. In essence the position adopted by nudge enthusiasts is that poor social outcomes are often the result of poor decision-making, and in order to address this, behavioural interventions such as nudges can be used to reduce the likelihood of poor decisions being made in the first place.

Dr Osman said: “Overall, the public make pretty sensible judgments, and what this shows is that people will scrutinize the information they are provided by experts, so long as they are given a means to do it. In other words, ask the questions in the right way, and people will show a level of scrutiny that is often not attributed to them. So, before there are strong claims made about public opinion about experts, and knee-jerk policy responses to this, it might be worth being a bit more careful about how the public are surveyed in the first place.”
Press reports:
  • The  Daily Record: Stirred by science:

Tuesday, 3 July 2018

How Bayesian Networks are pioneering the ‘smart data’ revolution

The July issue of Open Access Government has a 2-page article summarising our recent research and tool developments on Bayesian networks. A high-res pdf article of the article can be found here or here.



Thursday, 28 June 2018

Guilty Until Proven Innocent: The Crisis in Our Justice System



As mentioned in my previous posting I was invited by Jon Robins (the Justice Gap) to speak at the third meeting of the All-Party Parliamentary Group on Miscarriages of Justice, hosted by Barry Sheerman MP, in the House of Commons on 25 June 2018. The meeting was based around the launch of Jon Robins' outstanding new book "Guilty Until Proven Innocent: The Crisis in Our Justice System".  Other speakers were: Michael Mansfield QC and lawyer Matt Foot who have been involved in many of the cases described in the book; Waney Squier the world-renowned neuropathologist who suffered for being one of the few medical experts to question the mainstream medical guidelines on 'shaken baby syndrome'; Gloria Morrison who spoke about the problems of Joint Enterprise relevant to some of the cases; and Liam Allan and Eddie Gilfoyle who spoke about their own experiences (theirs are are two of the cases discussed in the book). It was a very powerful and informative meeting which was very well attended (with many having to stand for the full two hours)

I  have now written a detailed review of the book (also available here) which includes more about the House of Commons meeting.


See also



Monday, 25 June 2018

On the Role of Statistics in Miscarriages of Justice


I have been invited by Jon Robins (the Justice Gap) to speak today at the third meeting of the All-Party Parliamentary Group on Miscarriages of Justice, hosted by Barry Sheerman MP, in the House of Commons. Jon Robins will be talking about his outstanding new book "Guilty Until Proven Innocent: The Crisis in Our Justice System" at the event. The book includes a description of the Ben Geen case for which I provided a report to the Criminal Cases Review Commission in 2015 showing that the sequence of 'unusual events' at the Horton General Hospital (where Ben Geen worked as a nurse) was not especially unusual.

My short talk today focuses on the role of statistics in miscarriages of justice. A transcript of the talk can be found here.

Norman Fenton

See also

Friday, 22 June 2018

Bias in AI Algorithms


This is an update of a posting originally made on  18 Jan 2018 (see below for the update)

On 17 Jan 2018 multiple news sources (e.g. see here, here, and here) ran a story about a new research paper ‎ that claims to expose both the inaccuracies and racial bias in COMPAS - one of the most common algorithms used for parole and sentencing decisions to predict recidivism (i.e. whether or not a defendant will re-offend).

The research paper was written by the world famous computer scientist Hany Farid (along with a student Julia Dressel).

But the real story here is that the paper’s accusation of racial bias (specifically that the algorithm is biased against black people) is based on a fundamental misunderstanding of causation and statistics. The algorithm is no more ‘biased’ against black people than it is biased against white single parents, ‎ old people, people living in Beattyville Kentucky, or women called ‘Amber’. In fact, as we show in this brief article, if you choose any factor that correlates with poverty you will inevitably replicate the statistical ‘bias’ claimed in the paper. And if you accept the validity of the claims in the paper then you must also accept, for example, that a charity which uses poverty as a factor to identify and help homeless people is being racist because it is biased against white people (and also, interestingly, Indian Americans).

The fact that the article was published and that none of the media running the story realise that they are pushing fake news is what is most important here. Depressingly, many similar research studies involving the same kind of misinterpretation of statistics result in popular media articles that push a false narrative of one kind or another.

22 June 2018 Update: It turns out that now Microsoft is "developing a tool to help engineers catch bias in algorithms" This article also cites the case of the COMPAS software:
 "...., which uses machine learning to predict whether a defendant will commit future crimes, was found to judge black defendants more harshly than white defendants." 
Interestingly, this latest news article about Microsoft does NOT refer to the 2018 Dressel and Fardi article but, rather, to an earlier 2016 article by Larson et al: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm From a quick inspection it does seem to be a more comprehensive study than the flawed Dressel and Farid article. But my quick impression is that the same fundamental misunderstandings statistics/causality are there. Given the great degree of interest in AI/bias, and given also that we were unaware of the 2016 study, we plan to do an update to our unpublished paper.

Our article (5 pages): Fenton, N.E., & Neil, M. (2018). "Criminally Incompetent Academic Misinterpretation of Criminal Data - and how the Media Pushed the Fake News" http://dx.doi.org/10.13140/RG.2.2.32052.55680  Also available here.

The research paper: Dressel, J. & Farid, H. The accuracy, fairness, and limits of predicting recidivism. Sci. Adv. 4, eaao5580 (2018). 

Thanks to Scott McLachlan for the tip off on this story.

See some previous articles on poor use of statistics:

Wednesday, 20 June 2018

New project: Bayesian Artificial Intelligence for Decision Making under Uncertainty


Anthony Constantinou - a lecturer based in the Risk and Information Management Group at Queen Mary University of London - has been awarded a prestigious 3-year EPSRC Fellowship Grant £475,818 in partnership with Agena Ltd to develop open-source software that will enable end-users to quickly and efficiently generate Bayesian Decision Networks (BDNs) for optimal real-world decision-making. BDNs are Bayesian Networks augmented with additional functionality and knowledge-based assumptions to represent decisions and associated utilities that a decision maker would like to optimize. BDNs are suitable for modelling real-world situations where we seek to discover the optimal decision path to maximise utilities of interest and minimise undesirable risk.

A full description of the project can be found here. The EPSRC announcement is here.
 
Links

Thursday, 24 May 2018

The limitations of machine learning


Readers of this and our other blogs will be aware that we have long been sceptical of the idea that 'big data' - coupled with clever machine learning algorithms - will be able to achieve improved decision-making and risk assessment as claimed (see links below). We believe that a smart data approach that combines expert judgment (including understanding of underlying causal mechanisms) with relevant data is required and that Bayesian Networks (BNs) provide an ideal formalism for doing this effectively.

Turing award winner Judea Pearl, who was the pioneer of BNs, has just published a new book "The Book of Why: The New Science of Cause and Effect", which delivers essentially this same message. And there is a great interview with Pearl in the Atlantic Magazine about the book and his current views. The article includes the following:
As he sees it, the state of the art in artificial intelligence today is merely a souped-up version of what machines could already do a generation ago: find hidden regularities in a large set of data. “All the impressive achievements of deep learning amount to just curve fitting,” he said recently.
Read it all.

The interview also refers to the article "Human-Level Intelligence or Animal-Like Abilities?" by Adnan Darwiche. This is an outstanding paper (8 pages) that explains in more detail why we do not need to be over impressed by deep learning.

Links
Norman gets his hands on Pearl's new book
p.s. The second edition of our book:  Fenton, N.E. and M. Neil, "Risk Assessment and Decision Analysis with Bayesian Networks" (with foreword by Judea Pearl) will be available August 2018. See this space.

Friday, 4 May 2018

Anthony Constantinou's football prediction system wins second spot in international competition


Anthony Constantinou
QMUL lecturer Dr Anthony Constantinou of the RIM research group has come second in an international competition to produce the most accurate football prediction system. Moreover, the winners (whose predictive accuracy was only very marginally better) actually based their model on the previously published pi-ratings system of Constantinou and Fenton.





Anthony's model Dolores was developed for the International Machine Learning for Soccer Competition hosted by the Machine Learning journal.

All participants were provided with the results of matches from 52 different leagues around the world - with some missing data as part of the challenge. They had to produce a single model before the end of March 2017 that would be tested on its accuracy of predicting 206 future match outcomes from 26 different leagues, played from March 31 to April 9 in 2017.

Dolores was ranked 2nd with a predictive accuracy almost the same as the top ranked system (there was less than 1% error rate difference between the two; the error rate was nearly 120% lower than the participants ranked lowest among those that passed the basic criteria).

Dolores is  designed to predict football match outcomes in one country by observing football matches in multiple other countries.It is based on a) dynamic ratings and b) Hybrid Bayesian Networks.

Unlike past academic literature which tends to focus on a single league or tournament, Dolores provides empirical proof that a model can make a good prediction for a match outcome between teams 𝑥 and 𝑦 even when the prediction is derived from historical match data that neither 𝑥 nor 𝑦 participated in. This implies that we can still predict, for example, the outcome of English Premier League matches, based on training data from Japan, New Zealand, Mexico, South Africa, Russia, and other countries in addition to data from the English Premier league.

The Machine Learning journal has published the descriptions of the highest ranked systems in its latest issue published online today. The full reference for Anthony's paper is:

Constantinou, A. (2018). Dolores: A model that predicts football match outcomes from all over the world. Machine Learning, 1-27, DOI: https://doi.org/10.1007/s10994-018-5703-7

The full published version can be viewed (for free) at https://rdcu.be/Nntp. An open access pre-publication version (pdf format) is available for download here.

This work was partly supported by the European Research Council (ERC), research project ERC-2013-AdG339182-BAYES_KNOWLEDGE
The DOLORES Hybrid Bayesian Network was built and run using the AgenaRisk software.

The full reference for the pi-ratings model (used by the competition's winning team) is:
Constantinou, A. C. & Fenton, N. E. (2013). Determining the level of ability of football teams by dynamic ratings based on the relative discrepancies in scores between adversaries. Journal of Quantitative Analysis in Sports. Vol. 9, Iss. 1, 37–50. DOI: http://dx.doi.org/10.1515/jqas-2012-0036
Open access version here.
See also:

Tuesday, 17 April 2018

Explaining Bayesian Networks through a football management problem


Today's Significance Magazine (the magazine of the Royal Statistical Society and the American Statistical Association) has published an article by Anthony Constantinou and Norman Fenton that explains, through the use of an example from football management, the kind of assumptions required to build useful Bayesian networks (BNs) for complex decision-making. The article highlights the need to fuse data with expert knowledge, and describes the challenges in doing so. It also explains why, for fully optimised decision-making, extended versions of BNs, called Bayesian decision networks, are required.

The published pdf (open source) is also available here and here.

Full article details:
Constantinou, A., Fenton, N.E, "Things to know about Bayesian networks", Significance, 15(2), 19-23 April 2018, https://doi.org/10.1111/j.1740-9713.2018.0

Wednesday, 14 March 2018

Tuesday, 6 March 2018

Two coins: one fair one biased

Alexander Bogolmony tweeted this problem:


If there is no reason to assume in advance that either coin is more likely to be the coin tossed once (i.e. the first coin) then all the (correct) solutions show that the first coin is more likely to be biased with a probability of 9/17 (=0.52941). Here is an explicit Bayesian network solution for the problem:


The above figure shows the result after entering the 'evidence' (i.e. one Head on the coin tossed once and two Heads on the coin tossed three times). The tables displayed are the conditional probability tables defined for the associated with the variables.

This model took just a couple of minutes to build in AgenaRisk and requires absolutely no manual calculations as the Binomial distribution is one of many functions pre-defined. The model (which can be run in the free version of AgenaRisk is here). The nice thing about this solution compared to the others is that it is much more easily extendible. It also shows the reasoning very clearly.


Monday, 12 February 2018

An Improved Method for Solving Hybrid Influence Diagrams

Most decisions are made in the face of uncertain factors and outcomes. In a typical decision problem, uncertainties involve both continuous factors (e.g. amount of profit) and discrete factors (e.g. presence of a small number of risk events). Tools such as decision trees and influence diagrams are used to cope with uncertainty regarding decisions, but most implementations of these tools can only deal with discrete or discretized factors and ignore continuous factors and their distributions.

A paper just published in the International Journal of Approximate Reasoning presents a novel method that overcomes a number of these limitations. The method is able to solve decision problems with both discrete and continuous factors in a fully automated way. The method requires that the decision problem is modelled as a Hybrid Influence Diagrams, which is an extension of influence diagrams containing both discrete and continuous nodes, and solves it by using a state-of-the-art inference algorithm called Dynamic Discretization. The optimal policies calculated by the method are presented in a simplified decision tree.



The full reference is:

Yet, B., Neil, M., Fenton, N., Dementiev, E., & Constantinou, A. (2018). "An Improved Method for Solving Hybrid Influence Diagrams". International Journal of Approximate Reasoning. DOI: 10.1016/j.ijar.2018.01.006  Preprint (open access) available here.
UPDATE (22 Feb 2018): The full published version the paper is available online for free for 50 days here: https://authors.elsevier.com/c/1Wc6D,KD6ZG8y-

Acknowledgements: Part of this work was performed under the auspices of EU project ERC-2013-AdG339182-BAYES_KNOWLEDGE

Saturday, 10 February 2018

Decision-making under uncertainty: computing "Value of Information"


Information gathering is a crucial part of decision making under uncertainty. Whether to collect additional information or not, and how much to invest for such information are vital questions for successful decision making. For example, before making a treatment decision, a physician has to evaluate the benefits and risks of additional imaging or laboratory tests and decide whether to ask for them. Value of Information (VoI) is a quantitative decision analysis technique for answering such questions based on a decision model. It is used to prioritise the parts of a decision model where additional information is expected to be useful for decision making.

However, computing VoI in decision models is challenging especially when the problem involves both discrete and continuous variables. A new paper in the IEEE Access journal illustrates a simple and practical approach that can calculate VoI using Influence Diagram models that contain both discrete and continuous variables. The proposed method can be applied to a wide variety of decision problems as most decisions can be modelled as an influence diagram, and many decision modelling tools, including Decision Trees and Markov models, can be converted to an influence diagram.

The full reference is:

Yet, B., Constantinou, A., Fenton, N., & Neil, M. (2018). Expected Value of Partial Perfect Information in Hybrid Models using Dynamic Discretization.  IEEE Access. DOI: 10.1109/ACCESS.2018.2799527

Acknowledgements: Part of this work was performed under the auspices of EU project ERC-2013-AdG339182-BAYES_KNOWLEDGE, EPSRC project EP/P009964/1: PAMBAYESIAN, and ICRAF Contract No SD4/2012/214 issued to Agena.

Wednesday, 7 February 2018

Lawnmower v terrorist risk: the saga continues


Kim Kardashian's tweet comparing risk from lawnmowers v terrorists triggered the award and debate

Yesterday Significance Magazine (the magazine of the Royal Statistical Society and the American Statistical Association) published an article “Lawnmowers versus Terrorists”  with the strapline:
The Royal Statistical Society’s first ‘International Statistic of the Year’ sparked plenty of online discussion. Here, Norman Fenton and Martin Neil argue against the choice of winner, while Nick Thieme writes in support.
Our case, titled “A highly misleading view of risk”, was an edited version of a paper  previously publicised in a blog post that itself followed up on original concerns raised by Nicholas Nassim Taleb about the RSS citation and the way it had been publicised. The ‘opposing’ case made by Nick Thieme was essentially a critique of our paper.

We have today published a response to Nick’s critique.


Links:

Monday, 5 February 2018

Revisiting a Classic Probability Puzzle: the Two Envelopes Problem


Many people have heard about the Monty Hall problem. A similar (but less well known and more mathematically interesting) problem is the two envelopes problem, which Wikipedia describes as follows:
“You are given two indistinguishable envelopes, each containing money, one contains twice as much as the other. You may pick one envelope and keep the money it contains. Having chosen an envelope at will, but before inspecting it, you are given the chance to switch envelopes. Should you switch?”
The problem has been around in various forms since 1953 and has been extensively discussed (see, for example Gerville-Réache for a comprehensive analysis and set of references)  although I was not aware of this until recently.

We actually gave this problem (using boxes instead of envelopes) as an exercise in the supplementary material for our Book, after Prof John Barrow of University of Cambridge first alerted us to it. The ‘standard solution’ (as in the Monty Hall problem) says that you should always switch. This is based on the following argument:
If the envelope you choose contains $100 then there is an evens chance the other envelope contains $50 and an evens chance it contains $200. If you do not switch you have won $100. If you do switch you are just as likely to decrease the amount you win as increase it. However, if you win the amount increases by $100 and if you lose it only decreases by $50. So your expected gain is positive (rather than neutral). Formally, if the envelope contains S then the expected amount in the other envelope is 5/4 times X (i.e. 25% more).
In fact (as pointed out by a reader Hugh Panton), the problem with the above argument is that it equally applies to the ‘other envelope’ thereby suggesting we have a genuine paradox. In fact, it turns out that the above argument only really works if you actually open the first envelope (which was explicitly not allowed in the problem statement) and discover it contains S. As Gerville-Réache shows, if the first envelope is not opened, the only probabilistic reasoning that does not use supplementary information leads to estimating expectations as infinite amounts of each envelope. Bayesian reasoning can be used to show that there is no benefit in switching, but that is not what I want to describe here.

What I found interesting is that I could not find - in any of the discussions about the problem - a solution for the case where we assume there is a finite maximum prize, even if we allow that maximum to be as large as we like. With this assumption it turns out that we can prove (without dispute) that there is no benefit to be gained if you stick or switch. See this short paper for the details:
Fenton N E, "Revisiting a Classic Probability Puzzle: the Two Envelopes Problem" 2018, DOI10.13140/RG.2.2.24641.04960

Friday, 19 January 2018

Criminally Incompetent Academic Misinterpretation of Criminal Data - and how the Media Pushed the Fake News


On 17 Jan 2018 multiple news sources (e.g. see here, here, and here) ran a story about a new research paper ‎ that claims to expose both the inaccuracies and racial bias in one of the most common algorithms used by parole boards to predict recidivism (i.e. whether or not a defendant will re-offend).

The research paper was written by the world famous computer scientist Hany Farid (along with a student Julia Dressel).

But the real story here is that the paper’s accusation of racial bias (specifically that the algorithm is biased against black people) is based on a fundamental misunderstanding of causation and statistics. The algorithm is no more ‘biased’ against black people than it is biased against white single parents, ‎ old people, people living in Beattyville Kentucky, or women called ‘Amber’. In fact, as we show in this brief article, if you choose any factor that correlates with poverty you will inevitably replicate the statistical ‘bias’ claimed in the paper. And if you accept the validity of the claims in the paper then you must also accept, for example, that a charity which uses poverty as a factor to identify and help homeless people is being racist because it is biased against white people (and also, interestingly, Indian Americans).

The fact that the article was published and that none of the media running the story realise that they are pushing fake news is what is most important here. Depressingly, many similar research studies involving the same kind of misinterpretation of statistics result in popular media articles that push a false narrative of one kind or another.

Our article (5 pages): Fenton, N.E., & Neil, M. (2018). "Criminally Incompetent Academic Misinterpretation of Criminal Data - and how the Media Pushed the Fake News" http://dx.doi.org/10.13140/RG.2.2.32052.55680  Also available here.

The research paper: Dressel, J. & Farid, H. The accuracy, fairness, and limits of predicting recidivism. Sci. Adv. 4, eaao5580 (2018). 


See some previous articles on poor use of statistics:

Thursday, 11 January 2018

On lawnmowers and terrorists again: the danger of using historical data alone for decision-making

The short paper and blog posting we did last week generated a lot of interest, especially after Nicholas Taleb retweeted it. An edited version (along with a response from a representative of the Royal Statistical Society) is going to appear in the February issue of Significance magazine (which is the magazine of the RSS and the American Statistical Association). In the mean time we have produced another short paper that explores further problems with the 'lawnmower versus terrorist risk' statistics - in particular the inevitable limitations and dangers of relying on historical data alone for risk assessment:
Fenton, N.E., & Neil, M. (2018). "Is decision-making using historical data alone more dangerous than lawnmowers?", Open Access Report DOI:10.13140/RG.2.2.20914.71363. Also available here.

Wednesday, 3 January 2018

Are lawnmowers a greater risk than terrorists?

Kim Kardashian, whose tweet comparing the threats of lawnmowers and terrorists led to RSS acclaim
In December 2017 the Royal Statistical Society (RSS) announced the winner of its “International Statistic of the Year”. The statistic was simply "69" which it said was "the annual number of Americans killed, on average, by lawnmowers - compared to two Americans killed annually, on average, by immigrant Jihadist terrorists".  The full RSS citation says that the statistic tweeted by Kim Kardashian ‘highlights misunderstandings of risk’ and ‘illuminates the bigger picture’. Unfortunately, we believe it does exactly opposite as we explain in this brief paper:
Fenton, N.E., & Neil, M. (2018). "Are lawnmowers a greater risk than terrorists?" Open Access Report DOI:10.13140/RG.2.2.34461.00486/1 
As you can see from the tweet by Taleb, this use of statistics for risk assessment was not universally welcomed.


See update to this story here.