All you need to know to understand why the ONS data on mortality by vaccine status is systemically flawed

You don’t need to look very hard into the latest ONS data on all-cause mortality by vaccination status to confirm what we have been saying for a long time (see for example here and here): the ONS data is so obviously flawed (due to miscategorisation, missing vaccine deaths, and underestimates of the proportion of unvaccinated) to be worthless. 

This simple extract from Table 3 on the age standardized mortality rate (ASMR) *** for non-covid deaths over the full period (Jan 2021 – May 2022) is all you need to know to realise everything else in the report is problematic.

 

	age standardized mortality rate for non-covid deaths (per 100K people)
Unvaccinated	1474.3
Ever vaccinated	892.9

(note that the ONS have, curiously, not provided the combined population ASMR for either non-covid deaths or all deaths for this latest period - we have written to the authors of the report requesting these numbers since they cannot be computed from the data provided).

Since this is the rate for non-covid deaths it should - in both categories - be similar to the historical age standardized mortality rate in pre-Covid times. Well, it just so happens the ONS has produced exactly those figures:

Year	age standardized mortality rate
2019	925.0
2018	965.4
2017	965.3
2016	966.9
2015	993.2
2014	953.0
2013	985.9
2012	987.4
2011	978.6

Notice how stable those figures are year on year (even in the main 'covid' year 2020 the ASM only went up to 1043). So what could possibly explain a 60% increase in non-covid deaths in 2022 among the unvaccinated and a 5% drop in non-covid deaths among the vaccinated?

Even the most fervent vaccine supporters know that it is not a miracle cure for non-covid disease. Yet, if you believe the above ONS data, it must be - since the unvaccinated are dying at a much higher rate of non-covid illnesses.  We stopped doing detailed analyses of the ONS updates on this data in January when, having exposed the blatant flaws, we called on a) the ONS to publicly withdraw their dataset and b) the retraction of any claims made by others that are based upon it.  

In case there has been an error either in our understanding of Table 3 or in the ONS presentation of the data we will be happy to set the record straight and update this article. But the bizarre data in the above table is consistent with the bias we found in the previous ONS reports. This bias (whereby mortality rates for the unvaccinated are overestimated and mortality rates for the vaccinated are underestimated) will inevitably occur when people dying (of any cause) shortly after vaccination are classified as unvaccinated and where the proportion of unvaccinated is underestimated.

The above table exposes that, far from fixing the many problems (that we discussed in detail with the authors of the ONS reports) these systemic flaws may have got even worse. 

There was previously a laughable 'explanation' provided for the increase in non-covid mortality in the unvaccinated  - namely the 'healthy vaccinee effect'. We already debunked that both theoretically and empirically. In any case, even if such a ludicrous hypothesis was correct, it would still mean that all the data - and conclusions from it - would have to be adjusted to take acount of the bias (which they certainly have not done).

Anybody attempting to draw any conclusions from the ONS data, let alone do any intensive data analysis, are now wasting their time. While it is possible there may be some interesting trends to discover from the data (such as possibly noticed here), these are more likely to simply reveal the extent to which the misclassification problems and other flaws have changed over time.

One of the most depressing things about the last two years has been how silent almost the entire academic community of statisticians and mathematicians has been in the face of so many obviously flawed statistics/studies that promote the ‘official narrative’ on Covid and the vaccine. A small number have contacted us privately to say they see through the nonsense but cannot speak out because they know how damaging it will be to their careers. In contrast, some prominent statisticians and mathematicians, who have refused to call out obviously flawed data supporting the ‘Covid narrative’, have been vocal in trying to downplay/ridicule more reliable data that supports an alternative narrative. Perhaps some of the prominent statisticians might at least finally publicly acknowledge a problem with the ONS data since it is so obvious?

*** we have also complained about the fact that the ONS started to use the age standardized mortality rate (unnecessary when you have the detailed breakdown by age) since it is essentially just a model that may not be suitable in this context.

Response to Susan Oliver video “Antivaxxers fooled by p-hacking and apples to oranges comparison”

 

The video and the tweet publicising it

On 26 June 2022 Susan Oliver published a video on YouTube titled “Antivaxxers fooled by p-hacking and apples to oranges comparison” in response to a preprint [1] by 8 authors, one of whom was well-know BMJ Senior Editor Peter Doshi. She refers to the paper as the “Doshi paper” and we will use the same reference here even though Doshi is the last, rather than first, named author.  The paper demonstrates the increased risk of serious adverse events (SAEs) arising from the Pfizer and Moderna covid vaccine trials. Susan summarised her view of the paper in this tweet (which included the link to the video) that was retweeted by people like Prof Sir David Spiegelhalter (a world renowned expert on probability and risk) and Prof Peter Hansen (Econometrician, Data Scientist, and Latene Distinguished Professor of Economics at UNC, Chapel Hill):

 

What Susan says in the video and why it totally misrepresents the Doshi paper

Susan spends 3 minutes highlighting a number of people she refers to as “anti-vaxxers” who tweeted about the paper, including Jordan Peterson who she refers to as a "self-declared best-selling author" (note: his 2018 book sold over 3 million copies and was number 1 on Amazon).  Susan then states:

 “It’s basically just a rubbish paper that uses a technique known as p-hacking followed by some apples stuff oranges comparisons”.

Interestingly, despite the video title, Susan spends less than 30 seconds describing what p-hacking is and instead refers to a paper about it [2] (we agree entirely with the general concerns raised about p-hacking and show how it is avoided using Bayesian hypothesis testing [3]). But the key flaw in Susan's criticism is that the “Doshi paper” is not an example of p-hacking at all. They do not use p-values and, also contrary to the continued assertions of Susan, they make no claims at all of statistical significance. Rather, the paper provides risk differences and risk ratios with 95% confidence intervals (CIs) for the various different comparisons of vaccine v placebo. For example, here is their table of results for all serious adverse events (SAEs) and also of the subset of serious adverse events of special interest (serious AESIs):

If the authors had been “p-hacking” they would have chosen a p-value like 0.05 and would have added, for each comparison of vaccine v placebo, a ‘significance statistic’ and arrived at at least one example where the statistic was less than 0.05.  Then they would claim, for example, that the increased SAE rate was ‘significant’.  They do nothing like that at all.

Susan then claims that only by ‘combining’ the data from the different trials does Doshi get the (mythically claimed) ‘significant results’ and that such combining should simply not be done (this is one of her ‘apples and oranges’ comparison argument). But, while it is true that the paper does also look at the combined numbers for each class of SAE, it turns out that in each case, the risk ratio for the combined numbers is actually less than for the Pfizer trial alone. For example, for all SAEs the (median) risk ratio for Pfizer v placebo is 1.36 compared to just 1.15 for combined v placebo: the results are less, not more, ‘significant’. Our own Bayesian analysis of the results presented below makes this very clear.

Susan’s final criticisms of the Doshi paper concerns the selection of SAEs and the possibility of ‘double counting’. Regarding selection, the events included and not included are governed by the WHO endorsed Brighton scheme, and are not decided by the authors, so this is a critical error Susan makes. The Brighton list was created a priori, based on data before the any results were released from the trials. Any double counting, such as with the diarrhoea and abdominal pain example she uses, are a direct effect of the fact that the data are not public. There’s merit to both measures - counting number of participants (with any SAE) and number of events. If one person has two SAEs that is worse than one person having one SAE. “Double counting” sounds bad, but this is not double counting. Doshi et al are measuring how many SAEs occur in the vaccine group versus the placebo group. If Diarrhoea and abdominal pain were each recorded as a SAE, then that is two SAEs. We don’t know which ones were in the same person as Pfizer/Moderna have not released IPD. In any case, the authors recognise the issue that, because some SAEs occur in the same person, the SAEs are not all independent events; they note it in the paper, and introduce an adjustment to standard error to account for it. It is unclear whether the adjustment is sufficient, but it actually weakens their case (it increases the size of the confidence intervals) - so they can hardly be accused of bias. 

Further regarding double counting, SAEs are counted individually to avoid them being hidden. So, if you get renal failure and then your penis drops off that should be two SAEs, not one.  One person having three SAEs (renal failure, penis drops off, stroke) could be considered as serious as three people having a stroke; so, although some clinicians disagree, it is entirely reasonable to count SAEs separately.  But Susan does not appear to understand what a SAE is. She assumes something like diarrhoea cannot be a SAE because lots of diarrhoea happens to be mild. But most covid is not serious, either. So diarrhoea can be a SAE if it’s serious enough and meets the regulatory criteria. And it’s a leading cause of death in some places.

That addresses all the ‘flaws’ that Susan claims about the paper. It is also important to note that, even when all the SAEs in the Pfizer and Moderna trial are combined, the absolute risk increase is fairly small -  a fact already made clear by Doshi et al. (although this is to be balanced against the very low risks of severe covid, which is in essence the core message of the paper). They state that, in this case, the absolute risk increase (95% CI) is between 2.1 to 22.9 events per 10,000 participants.  In our Bayesian analysis the median absolute risk increase is 12.9 events per 10,000 participants with CI between 0 to 27.

What it also ironic about the attack on the Doshi paper is that, just before her concluding remarks and ball juggling, Susan uncritically cites a very flawed modelling study crediting nearly 20 millions of lives saved to Covid jabs as a "rebuttal."

Bayesian analysis of the data

The benefits of applying a Bayesian analysis to the data is that we are able to ‘learn’ the full probability distributions of the adverse reaction rates for vaccine and placebo. This enables us not just to compute the risk ratios and CIs (we get slightly different results to Doshi) but, crucially, also to make explicit probabilistic statements about whether the vaccine SAE rate is higher than that of the placebo (this approach is the Bayesian alternative to the flawed p-value approach). The results (which we provide below) do indeed provide explicit support for the hypothesis that the SAE rate for vaccine is higher than that of the placebo.

 Serious adverse events

	Vacc AEs	Placebo AEs	P(Vacc AE) Median & CI	P(placebo AE) Median & CI	RR Median & CI	Prob (vacc higher SAE)
Pfizer	127/18801	93/18785	0.0068 (0.0057, 0.0080)	0.0050 (0.0040, 0.0060)	1.362 (1.044, 1.784)	98.86%
Moderna	206/15185	196/15166	0.01361 (0.0118, 0.0155)	0.01297 (0.0112, 0.0148)	1.050 (0.864, 1.275)	68.76%
Combined	333/33986	289/33951	0.0098 (0.0088, 0.0109)	0.0085 (0.0076, 0.0096)	1.151 (0.983, 1.348)	96.03%

 

Serious adverse events of special interest

	Vacc AEs	Placebo AEs	P(Vacc AE) Median & CI	P(placebo AE) Median & CI	RR Median & CI	Prob (vacc higher SAE)
Pfizer	52/18801	33/18785	0.00280 (0.0021, 0.0036)	0.00180 (0.0013, 0.0025)	1.56 (1.016, 2.44)	97.92%
Moderna	87/15185	64/15166	0.0058 (0.047, 0.0071)	0.0043 (0.0033, 0.0054)	1.37 (0.98, 1.88)	96.85%
Combined	139/33986	97/33951	0.0041 (0.0035, 0.0048)	0.0029 (0.0023, 0.0035)	1.43 (1.104, 1.857)	99.65%

 

Serious adverse events of special interest matching Brighton’s list

	Vacc AEs	Placebo AEs	P(Vacc AE) Median & CI	P(placebo AE) Median & CI	RR Median & CI	Prob (vacc higher SAE)
Pfizer	39/18801	28/18785	0.0021 (0.0015, 0.0028)	0.0015 (0.0010, 0.0022)	1.38 (0.86, 2.26)	90.82%
Moderna	65/15185	56/15166	0.0043 (0.0035, 0.0048)	0.0037 (0.0029, 0.0048)	1.16 (0.81, 1.66)	79.04%
Combined	104/33986	84/33951	0.00308 (0.0025, 0.00371)	0.00249 (0.002, 0.0031)	1.24  (0.93, 1.65)	92.57%

 

 

Postscrip: the vicious campaign against Peter Doshi

Following a tweet by Norman Fenton criticising the video blue checkmark ‘surgeon/scientist’  David Gorski made several replies supporting the claims of the video and then made this attack of Peter Doshi.

 

to which blue checkmark Steve Salzberg (“Bloomberg Distinguished Professor of BME, CS, and Biostats at Johns Hopkins University”) replied:

 

 and was supported by Art Caplan - Professor of bioethics:

 

 

References

[1] Fraiman, J., Erviti, J., Jones, M., Greenland, S., Whelan, P., Kaplan, R. M., & Doshi, P. (2022). Serious Adverse Events of Special Interest Following mRNA Vaccination in Randomized Trials. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4125239

[2] Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD (2015) The Extent and Consequences of P-Hacking in Science. PLoS Biol 13(3): e1002106. https://doi.org/10.1371/journal.pbio.1002106

[3] “A simple example of Bayesian hypothesis testing”, https://youtu.be/s4yCu__18Jo