The limitations of machine learning

Readers of this and our other blogs will be aware that we have long been sceptical of the idea that 'big data' - coupled with clever machine learning algorithms - will be able to achieve improved decision-making and risk assessment as claimed (see links below). We believe that a smart data approach that combines expert judgment (including understanding of underlying causal mechanisms) with relevant data is required and that Bayesian Networks (BNs) provide an ideal formalism for doing this effectively.

Turing award winner Judea Pearl, who was the pioneer of BNs, has just published a new book "The Book of Why: The New Science of Cause and Effect", which delivers essentially this same message. And there is a great interview with Pearl in the Atlantic Magazine about the book and his current views. The article includes the following:

As he sees it, the state of the art in artificial intelligence today is merely a souped-up version of what machines could already do a generation ago: find hidden regularities in a large set of data. “All the impressive achievements of deep learning amount to just curve fitting,” he said recently.

Read it all.

The interview also refers to the article "Human-Level Intelligence or Animal-Like Abilities?" by Adnan Darwiche. This is an outstanding paper (8 pages) that explains in more detail why we do not need to be over impressed by deep learning.

Links

• How a Pioneer of Machine Learning Became One of Its Sharpest Critics
• Pearl: "The Book of Why"
• Adnan Darwiche's article "Human-Level Intelligence or Animal-Like Abilities?" 
• Pearl: "Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution"
• The problem with big data and machine learning 
• A short story illustrating why pure machine learning (without expert input) may be doomed to fail and totally unnecessary (2 page pdf)
• Another machine learning fable: explains why pure machine learning for identifying credit risk may result in perfectly incorrect risk assessment (1 page pdf)
• Moving from big data and machine learning to smart data and causal modelling: a simple example from consumer research and marketing (7 page pdf)

Norman gets his hands on Pearl's new book

p.s. The second edition of our book:  Fenton, N.E. and M. Neil, "Risk Assessment and Decision Analysis with Bayesian Networks" (with foreword by Judea Pearl) will be available August 2018. See this space.

Probability and Statistics in Forensics

Never got round to posting this summary (it's the image above- click to enlarge) of our Isaac Newton Institute Cambridge Programme on Probability and Statistics in Forensic Science. The full version of the annual 2016-17 Report is here.

See also:

• Bayesian Networks and Argumentation in Evidence Analysis
• Bayes and the Law: what's been happening in Cambridge and how you can see it
• Recommendations for Dealing with Quantitative Evidence in Criminal Law 
• Bayes and the Law: Cambridge event and new review paper
• Confusion over the Likelihood ratio
• The Bayesian networks mutual exclusivity problem
• Use of Bayes in the Netherlands Court of Appeal

Anthony Constantinou's football prediction system wins second spot in international competition

Anthony Constantinou

QMUL lecturer Dr Anthony Constantinou of the RIM research group has come second in an international competition to produce the most accurate football prediction system. Moreover, the winners (whose predictive accuracy was only very marginally better) actually based their model on the previously published pi-ratings system of Constantinou and Fenton.





Anthony's model Dolores was developed for the International Machine Learning for Soccer Competition hosted by the Machine Learning journal.

All participants were provided with the results of matches from 52 different leagues around the world - with some missing data as part of the challenge. They had to produce a single model before the end of March 2017 that would be tested on its accuracy of predicting 206 future match outcomes from 26 different leagues, played from March 31 to April 9 in 2017.

Dolores was ranked 2nd with a predictive accuracy almost the same as the top ranked system (there was less than 1% error rate difference between the two; the error rate was nearly 120% lower than the participants ranked lowest among those that passed the basic criteria).

Dolores is  designed to predict football match outcomes in one country by observing football matches in multiple other countries.It is based on a) dynamic ratings and b) Hybrid Bayesian Networks.

Unlike past academic literature which tends to focus on a single league or tournament, Dolores provides empirical proof that a model can make a good prediction for a match outcome between teams 𝑥 and 𝑦 even when the prediction is derived from historical match data that neither 𝑥 nor 𝑦 participated in. This implies that we can still predict, for example, the outcome of English Premier League matches, based on training data from Japan, New Zealand, Mexico, South Africa, Russia, and other countries in addition to data from the English Premier league.

The Machine Learning journal has published the descriptions of the highest ranked systems in its latest issue published online today. The full reference for Anthony's paper is:

Constantinou, A. (2018). Dolores: A model that predicts football match outcomes from all over the world. Machine Learning, 1-27, DOI: https://doi.org/10.1007/s10994-018-5703-7

The full published version can be viewed (for free) at https://rdcu.be/Nntp. An open access pre-publication version (pdf format) is available for download here.

This work was partly supported by the European Research Council (ERC), research project ERC-2013-AdG339182-BAYES_KNOWLEDGE
The DOLORES Hybrid Bayesian Network was built and run using the AgenaRisk software.

The full reference for the pi-ratings model (used by the competition's winning team) is:

Constantinou, A. C. & Fenton, N. E. (2013). Determining the level of ability of football teams by dynamic ratings based on the relative discrepancies in scores between adversaries. Journal of Quantitative Analysis in Sports. Vol. 9, Iss. 1, 37–50. DOI: http://dx.doi.org/10.1515/jqas-2012-0036
Open access version here.

See also:

•  Anthony's pi-football website.
• Explaining and predicting football results over an entire season
• Explaining Bayesian networks through a football management problem
• The problem with predicting football results
• A Bayesian network to determine optimal strategy for Spurs' success
• Proving referee bias with Bayesian networks