Thursday 26 March 2015

The risk of flying

I have just done an interview on BBC Radio Scotland about aircraft safety in the light of the GermanWings crash - which now appears to have been a deliberate act of sabotage by the co-pilot*. I have uploaded a (not very good) recording of it here (mp3 file - it is just under 4 minutes) or here (a more compact m4a file)

Because this type of event is so rare classical frequentist statistics provides no real help when it comes to risk assessment. In fact, it is exactly the kind of risk assessment problem for which you need causal models and expert judgement (as explained in our book) if you want any kind of risk insights.

Irrespective of this particular incident, the interview gave me the opportunity to highlight a very common myth, namely that “flying is the safest form of travel”.  If you look at deaths per million travellers then, indeed, there are 50 times as many car deaths as plane deaths. However, this is a silly measure because there are so many more car travellers than plane travellers. So, typically, analysts use deaths per million miles travelled; with respect to this measure car travel is still 'riskier' than air travel, but the death rate is only about twice as high as plane deaths. But this measure is also biased in favour of planes because the average plane journey is much further than the average car journey.

So a much fairer measure is the number of deaths per passenger journey. And for this, the rate of plane deaths is actually three times higher than car deaths; in fact only bikes and motorbikes are worse than planes.

Despite all this there is still a very low probability of a plane journey resulting in fatalities - about 1 in half a million (and much less on commercial flights in Western Europe). However, if we have reason to believe that, say, recent converts to a terrorist ideology have been training and becoming pilots then the probability of the next plane journey resulting in fatalities becomes much higher, despite the past data.

*I had an hour’s notice of the interview and was told what I would be asked.  I was actually not expecting to be asked about how to assess the risk of this specific type of incident;  I was assuming I would only be asked about aircraft safety risk in general and about the safety record of the A320.

Postscript: Following the interview a colleage asked:
"Did you have the mental issues of the co-pilot on the radar when you replied? "
My response: Interesting question. A few years back we were involved extensively in work with NATS (National Air Traffic Safety) to model/predict risk of mid-air collision over the UK airspace. In particular NATS wanted to know how the probability of a mid-air collision might change given different proposals for changes to the ATM architecture (e.g. ‘adding new ground radar stations’ versus ‘adding new on-board collisions alert systems’). Now - apart from three incidents in the late 1940’s which all involved at least one military jet - there has not been any actual mid-air collisions over UK airspace (so negligible data there) and the proposed technology was ‘new’ (so no directly relevant data there) but there was a LOT of data on "near misses" of different degrees of seriousness  and a LOT of expert judgment about the causes and circumstances of the near misses. Hence, we were able with NATS experts to build a very detailed model that could be ‘validated’ against the actual near miss data. What is very interesting are what factors NATS needed in the model. The psychological state and stress of air traffic controllers was included in the model as were certain psychological traits of pilots. It turns out that certain airlines were more likely to be involved in a near-misses primarily because of traits of their pilots.

Tuesday 24 March 2015

The problem with big data and machine learning

The advent of ‘big data’, coupled with fancy statistical machine learning techniques, is increasingly seducing people to believe that new insights and better predictions can be achieved in a wide range of important applications, without relying on the input of domain experts. The applications range from learning how to retain customers through to learning what makes people susceptible to particular diseases. I have written before about the dangers of this kind of 'learning' from data alone (no matter how 'big' the data is).

Contrary to the narrative being sold by the big data community, if you want accurate predictions and improved, decision-making then, invariably, you need to incorporate human knowledge and judgment. This enables you to build rational causal models based on 'smart' data. The main objections to using human knowledge - that it is subjective and difficult to acquire - are, of course,  key drivers of the big data movement. But this movement underestimates the typically very high costs of collecting, managing and analysing big data. So, the sub-optimal outputs you get from pure machine learning do not even come cheap.

To clarify the dangers of relying on big data and machine learning, and to show how smart data and causal modelling (using Bayesian networks) gives you better results, I have collected together the following short stories and examples:
The whole subject of 'smart data' rather than 'big data' is also the focus of the research project BAYES-KNOWLEDGE.