14 month-old evidence submitted to UK Parliament on Covid-19 data transparency and accountability

Just discovered the statement below on the UK Government website (I’d totally forgotten about it). It’s the evidence we provided 14 months ago concerning data transparency and accountability during the COVID-19 crisis. A pdf version is here. 

Shame the Government totally ignored the evidence.

A Response to the Call for Evidence
Regarding COVID-19 Data
Transparency and Accountability

 

To: UK Parliament, Public Administration and Constitutional Affairs Committee, Commons Select Committee

 

1 November 2020

 

Professor Norman Fenton

Director of Risk Information Management Research Group

n.fenton@qmul.ac.uk

 

Professor Martin Neil

Professor in Computer Science and Statistics

 

Dr Scott McLachlan

Postdoctoral Researcher in Health Informatics (Queen Mary)

Fellow in Law (Birmingham Law School)

 

To:      UK Parliament

Public Administration and Constitutional Affairs Committee

Commons Select Committee

 

Honourable Members;

 

Re:      Data Transparency and Accountability: COVID-19.

 

Apropos the call for evidence concerning data transparency and accountability during the COVID-19 crisis.

We respond;

1.     We are a group of senior researchers in risk assessment, probability, statistics, and public health technologies based at Queen Mary University of London. Since March 2020, we have produced 23 articles/reports[1] (of which 5 have been published in peer reviewed journals) analysing the publicly available COVID-19 statistics and producing risk assessments and models.

2.     We believe that the statistics provided to and by the Government during the COVID-19  crisis have been inadequate and have been too easily used by influencers and decision-makers to fit particular narratives that have exaggerated the scale of the crisis.

3.     Statistics and data are observed phenomena arising from unobserved processes and their interactions (including causal explanations) as shown in Figure 1. The number of observed COVID-19 ‘cases’ clearly depends on how a ‘case’ is defined and the population infection rate, but it is also influenced by many (normally unreported) causal factors such as how many tests are being performed, who is being tested and why, and the accuracy of the testing. Similarly, while the number of observed COVID-19  ‘deaths’ clearly depends on how a COVID-19  death is defined and reported, it is also influenced by the population demographics, quality of healthcare etc. Hence, contrary to popular conception, data do not ‘speak for themselves’.

4.     For example, in March and April (as we pointed out in[2]^,[3],[4]), by focusing only on simple counts of ‘cases’, ‘hospitalisations’ and ‘deaths’, the public was misled into believing that the virus was more deadly than it really was. At that stage testing was essentially limited to those who were either already hospitalized with severe symptoms or were frontline healthcare workers. The reported high death (and hospitalisation) rates of those infected (calculated by simply dividing the number of deaths by the number of ‘cases’) were in part explained by the limited testing regime that was essentially only ‘finding’ the most severe ‘cases’.

5.     Similarly, the scale of the ‘second wave’ has been continually exaggerated by focusing on increased ‘cases’ without considering the simple causal explanation of massively increased testing. When this is done – as shown in the plots of Figure 2 –the trends for cases, hospitalizations and deaths look far less worrying than those presented at https://coronavirus.data.gov.uk using exactly the same data.

6.     At the root of the data problem there has been a fundamental misunderstanding about the meaning of terms ‘COVID-19 cases’ and ‘COVID-19 deaths’, and what can be interpreted from statistics that use these terms.  Even small changes in how these are defined and classified (as has happened several times since March) lead to very different trends and conclusions.

7.     The definition of a COVID-19 ‘case’ is especially concerning. In epidemiology, a case definition includes criteria for person (e.g. gender, race, age, or exclusion criteria), place (such as that associated with the outbreak of a disease), time (when illness started) and clinical features. Clinical features are initially normally simple and objective such as ‘sudden onset of fever and cough’ but should later be characterised by confirmed presence of specific laboratory findings, such as ‘ground-glass opacity on Chest CT and positive culture for SARS-CoV-2’. During this crisis, a positive PCR test has improperly become the surrogate replacing all four aspects of case definition. A PCR test may be positive: (i) before clinical features arise; (ii) long after clinical features have abated; or even (iii) when a person has simply come into contact with the disease but without them ever becoming infected. Some argue that reporting cycle threshold (Ct) values may help clinical decision-makers identify at which of these three stages an asymptomatic person may present; however, given that almost all so-called asymptomatic cases never develop active disease, if we leave aside issues with false positives (which increase for high Ct values), we submit that many ‘cases’ must be type (iii) and therefore did not meet the normal epidemiological standard to be classified or counted as a case.

8.     Confusion about the definition of a COVID-19 ‘death’ also persist. It is now clear that Government-reported deaths include not just those who died as a direct result of the disease, but also all of those who have died ‘with it’, thus leading to inflation of the fatality figures. Several studies have also suggested that reported deaths from other pneumonias, influenzas and even lung cancer have dropped well below normal annual levels since March. As such there are questions surrounding whether people who died of these similar conditions were incorrectly classified as COVID-19 deaths.

9.     With the massive increase in testing since August, uncertainty about the testing accuracy -  especially the false positive rate of PCR tests[5]   - means that almost nothing meaningful can be concluded about the increasing cases or fatality rate – see Figure 3.  The vast majority tested have no symptoms at all, so in the absence of data provided about the proportion of asymptomatic people who were tested and tested positive (as well as the other missing information shown in Figure 3), we do not know what proportion of new ‘cases’ and reported ‘deaths’ are people infected with COVID-19  at all. A false positive rate of even just 1% would, together with the massively increased testing, provide a causal explanation for the increase in cases even if the virus has largely subsided[6]. But, yet again, the narrative presented – and the one on which lockdown decisions are based – is that of a massive ‘second wave’.

10.  Closed loop thinking means that, once a particular narrative is ‘believed’, alternative explanations for the observed data are never entertained. Indeed, the lack of data, unscientific closed-shop models, fundamental misunderstandings by decision-makers, manipulation of underlying reporting processes, contradictory goals or the potential for malign intent are all feasible explanations for the observed data and chaotic analysis. The lack of data transparency gives credence to these explanations and leads to a lack of trust in government statistics and decisions made using those statistics.

11.  There are many examples of how the crude data, and failure to consider alternative causal explanations, has been used for inappropriate decision-making and even scare-mongering. These include:

i.      Using ‘100 new cases per 100,000 people’ as a threshold beyond which a local borough is required to move to lockdown. With this metric the threshold can be avoided or reached simply by decreasing/increasing the number of tests carried out.

ii.     As explained above (and in Figure 2) the headline figures and graphs – as presented for example, at https://coronavirus.data.gov.uk/ do not factor in the increase in testing. For example, the recent ‘exponential’ increase in number of cases – which has driven the ‘second wave narrative’ does not look at all serious when we plot it as number of cases per 1000 tests. The same is true of hospital admissions and deaths; for example,  contrary to the frightening ‘absolute’ increase in hospital cases since September, it turns out that the number of hospital admissions per 1000 cases has remained stable – and may even be decreasing when we factor in the false positives and those admitted for non-COVID reasons who happen to get a positive COVID test after admission.

iii.   The ONS report[7] on COVID-19 deaths by ethnicity is one of many that have produced misleading conclusions without even revealing all relevant data. This particular report exaggerated[8] the increased risk to people from the BAME community by using ‘relative risk’ to summarise the findings, rather than ‘absolute risk’ as continually  recommended for communicating risk to the public, by Royal Statistical Society Chairman (and member of SAGE) Professor Sir David Spiegelhalter[9]. Moreover, we noted[10]^, that the claims were almost certainly further exaggerated as they were likely based on out of date demographic information (the ONS failed to respond to our request to identify what data were used). Hence the ONS report – which was widely quoted in the media – was likely to create an unjustified level of fear and anxiety among the BAME community. Failure to identify causal explanations for data bias has also led to multiple well-publicised studies with exaggerated[11]^,[12] - or even flawed[13] -  claims that certain communities, or people with certain attributes or habits, are at much higher risk of COVID-19.

iv.   In early October news broke of under-reporting of almost 16,000 positive PCR tests and that, as a result, as many as 48,000 people may not have been informed of their exposure due to close contact with these undisclosed ‘cases’[14]. PHE blamed Microsoft’s Excel software[15], but this disingenuous admonition did more to highlight PHE’s: (a) reliance on almost 25yr old technology; (b) ignorance of and failure to maintain pace with technology; and (c) lack of any reliable approach to checking and validating data they collect and report. Data security experts describe this as one in a long string of data and information security failings by PHE and the Government and have used it to support eschewing use of the proposed NHSx track and trace apps[16].

v.     Removal or sanitising of flu incidence/death data from 1999 and all previous years from the ONS website making comparisons almost impossible and giving the impression that the ‘past is being rewritten or expunged’.

vi.   Constant changing of scales and metrics used in data reporting. For example, deaths were recorded as COVID-19 deaths if they occurred within 28 days of a positive test and this has recently been changed to 60 days if COVID-19 appears on the death certificate. This change was done in reaction to a recommendation that the period should be reduced to 21 days. The change was made with no accompanying   explanation of why it was increased rather than decreased[17].

12.  Ultimately the only way to achieve accurate estimates of the critical population infection rate at any given time is to provide the missing – but easy to obtain data shown in Figure 4.

13.  Decisions about lockdown require data to support the evidence shown in  Figure 5. If these data have been considered in Government decisions, they have certainly not been made public.

 

In summary, and supported by the arguments above our responses to the eight issues identified in the public call are:

 

14.  In response to Issue 1: Did the Government have good enough data to make decisions in response to Coronavirus, and how quickly were the Government able to gather new data?

Data provided by several departments including PHE, NHS and ONS for Government decision-making was observed to be ever-changing, unreliable, and of such poor quality and so inappropriately framed as to be insufficient to support the public health, policy and legislative decisions that resulted.

15.  In response to Issue 2: Was data for decision-making sufficiently joined up across departments?

Where multiple actors are responsible for collecting and reporting data that will be aggregated and used to direct public policy: definitions, thresholds and processes must observe a consistent standard. The central aggregator, in this case the ONS, should have been responsible for both dictating and enforcing that standard.

As evidenced by anomalies and misrepresentations identified above, the efforts of PHE, NHS and ONS were not sufficiently joined up, fell short of due standards, and severely undermined Government decision-making, independent scrutiny, and ultimately public confidence.

16.  In response to Issue 3: Was relevant data disseminated to key decision-makers in: Central and Local Government; other public services (like schools); businesses; and interested members of the public?

To be relevant, data must be capable of informing the decision-making process. Relevant data is that which is accurate, timely, indisputable, optimised and fit to inform the known purposes for which it may be used.

Given that Government was aware most members of the public consume only limited ‘views’ of such data as are presented in the media, its presentation, accuracy and fitness for purpose should have received greater consideration. While data was made available via the ONS website, for the reasons discussed above the relevance of this data has remained questionable.

17.  In response to Issue 4: Were key decisions (such as ‘lockdowns’) underpinned by good data and was data-led decision-making timely, clear and transparently presented to the public?

Government decisions impacting the liberty and freedoms of individuals appear to have been made haphazardly. While each came supported by justifications, it was claimed, that they were ‘led by the science’, more often it could be argued this was not the case.

Decision making has been presented as being the result of “the science” with the goal of delivering ‘consensus’. However, science does not operate as a consensus making mechanism and it is not monolithic. The current crisis has demonstrated that groups like SAGE and the Joint Biosecurity Centre are not following scientific norms of behaviour. Analysis and policy formulation need more stringent oversight in a way that invites and delivers scientific debate from both within and also outside the group.

18.  In response to Issue 5: Was data shared across the devolved administrations and local authorities to enable mutually beneficial decision-making?

If this were the case it has not been made clear to the public, and in any case, it is likely that the shared data suffered from all of the limitations we have highlighted.

19.  In response to Issue 6: Is the public able to comprehend the data published during the pandemic. Is there sufficient understanding among journalists and parliamentarians to enable them to present and interpret data accurately, and ask informed questions of Government?

It is difficult to ensure accurate comprehension in circumstances where, as discussed earlier, relevantly framed data has not been provided. Continued reliance on journalists to identify meaning from data has only resulted in sensational headlines that amplified public ignorance and promulgated fear.

What could have been done to improve understanding and who could take responsibility for this?

The current crisis has demonstrated Government must take additional steps to provide context and meaning capable of supporting differing interpretations they wish the public should draw from published data. The public should be trusted to understand nuance and scientific disagreement about what the data might be telling them.

20.  In response to Issue 7: Does the Government have a good enough understanding of data security, and do the public have confidence in the government’s data handling?

The policies and approaches of Government do not seem to have reflected prevailing opinions and wishes of the public. This has never been more obvious than during development and release of both versions of the NHSx Track and Trace smartphone app, and when vision-based population proximity monitoring AI systems were deployed around London suburbs, and once exposed in the media, hastily removed. It seems there is little public confidence in the current approach to securing public and personal data and indeed the potential for increased suspicion of the government’s motives in this regard.

21.  In response to Issue 8: How will the change in responsibility for Government data impact future decision-making?

It is not clear what the change in responsibility is and the motivation for it. Any change in responsibility might simply be akin to ‘rearranging the deckchairs on the Titanic’.

 

 

 

 

FIGURES

 

 

Figure 1 Causal model explaining observed data

 

Figure 2 Simple plots that take account of number of tests and cases

 

Figure 3 Why the daily reported data tell us almost nothing

Figure 4 Missing data needed to accurate estimation of population infection rate

 

 

Figure 5 The evidence we need to demonstrate why lockdowns are needed

 

---

[1] https://www.eecs.qmul.ac.uk/~norman/all_publications.htm#Papers

[2] Fenton, N. E., Neil, M., Osman, M., & McLachlan, S. (2020). "COVID-19 infection and death rates: the need to incorporate causal explanations for the data and avoid bias in testing". Journal of Risk Research, 1–4. https://doi.org/10.1080/13669877.2020.1756381

[3] Fenton, N. E., Osman, M., Neil, M., & McLachlan, S. (2020). Coronavirus: country comparisons are pointless unless we account for these biases in testing. The Conversation,  April 2, 2020  https://theconversation.com/coronavirus-country-comparisons-are-pointless-unless-we-account-for-these-biases-in-testing-135464

[4] Fenton, N.E., Hitman, G. A., Neil, M., Osman, M., & McLachlan, S. (2020). Causal explanations, error rates, and human judgment biases missing from the COVID-19 narrative and statistics. PsyArXiv Preprints. https://doi.org/10.31234/OSF.IO/P39A4

[5] Cohen, A. N., Kessel, B., & Milgroom, M. G. (2020). Diagnosing COVID-19 infection: the danger of over-reliance on positive test results. MedRxiv, 2020.04.26.20080911. https://doi.org/10.1101/2020.04.26.20080911

[6] https://probabilityandlaw.blogspot.com/2020/09/the-impact-of-covid-false-positives.html

[7] Office for National Statistics. (2020a). Coronavirus (Covid-19) related deaths by ethnic group, England and Wales -2 March 2020 to 10 April 2020.

[8] Fenton N. E, Neil M, McLachlan S, Osman M (2020), "Misinterpreting statistical anomalies and risk assessment when analysing Covid-19 deaths by ethnicity". DOI: 10.13140/RG.2.2.18957.5680

[9] https://www.regulation.org.uk/library/2017-Spiegelhalter-Risk_and_Uncertainty_Communication.pdf

[10] Fenton N. E, Neil M, McLachlan S, Osman M (2020), "Misinterpreting statistical anomalies and risk assessment when analysing Covid-19 deaths by ethnicity". DOI: 10.13140/RG.2.2.18957.5680

[11] Fenton N. E, Neil M, McLachlan S, Osman M (2020), "Misinterpreting statistical anomalies and risk assessment when analysing Covid-19 deaths by ethnicity". DOI: 10.13140/RG.2.2.18957.5680

[12] Fenton, N E. (2020). A Note on UK Covid19 death rates by religion: which groups are most at risk? http://arxiv.org/abs/2007.07083

[13] Fenton, N E (2020), "Why most studies into COVID19 risk factors may be producing flawed conclusions-and how to fix the problem", http://arxiv.org/abs/2005.08608 

[14] https://www.theregister.com/2020/10/05/excel_england_coronavirus_contact_error/

[15] https://www.bbc.co.uk/news/technology-54423988

[16] https://www.express.co.uk/comment/expresscomment/1344766/NHS-app-NHS-covid-19-contact-tracing-app-download-give-blood

[17] https://www.gov.uk/government/news/new-uk-wide-methodology-agreed-to-record-covid-19-deaths