Thursday 23 September 2021

A comparison of age adjusted all-cause mortality rates in England between vaccinated and unvaccinated

Norman Fenton and Martin Neil

See 26 October UPDATE to this article

The UK Government's own data does not support the claims made for vaccine effectiveness/safety. 

In a previous post we argued that the most reliable long-term measure of Covid-19 vaccine effectiveness/safety is the age adjusted all-cause mortality rate. If, over a reasonably prolonged period, fewer vaccinated people die, from whatever cause, including Covid-19, than unvaccinated people then we could conclude that the benefits of the vaccine outweigh the risks. We also pointed out that, to avoid the confounding effect of age, it is critical that data for each age category is available, rather than the aggregated data because, clearly, aggregated data might exaggerate vaccine mortality rates if more older people, with shorter expected mortality, are included. The UK roll out of the vaccine was executed in descending age order, from older to younger, except very early on in the vaccination programme when the vulnerable young were vaccinated along with the very elderly. As the programme progressed those vaccinated were, on average, older than those who remained unvaccinated and as the roll out proceeded a progressively higher proportion of the residual unvaccinated population are younger.

The  latest Office for National Statistics report on mortality rates by Covid vaccination status provides data on all deaths – Covid related and non-Covid related for the period Jan-July 2021 for the unvaccinated and the different categories of vaccinated ('within 21 days of first dose', '21 days or more after first dose', 'second dose'). The ONS data for Covid-19 mortality, is given in Table 4 of the ONS spreadsheet and the ONS data for all-cause mortality excluding Covid-19, is given in Table 5 of the same spreadsheet. Both tables are reproduced at the bottom of this post.

We believe there are severe weaknesses and possible errors in the ONS data (see foonote**). But importantly, while it does not provide the raw age categorized data, it does provide "age standardized" mortality rates*** (also see explanatory video). This means the ONS have calculated the overall mortality rate in a way which (they believe) adjusts for the confounding effect of age, and this is ‘baked into’ the mortality rates they have published.  However, while they report this age adjusted mortality rate for each of the three separate categories of vaccinated people they do not report it for the combined set of vaccinated people. In our analysis, and in the absence of the actual age stratified data, we compute a population weighted age adjusted all-cause mortality rate by using the ONS’s published population sizes for each of the three categories of vaccinated. This is not ideal because the ONS age adjusted rates are so opaque and are not 'abolute numbers'. However, in the absence of detailed data this should provide a reasonable estimate of what the ONS age adjusted all-cause mortality rate would be for all unvaccinated if they had bothered to report it. We will call this the ‘weighted vaccinated mortality rate’. The data table derived from the ONS data and used to compute this rate is given at the end of this post.

It turns out that, even using this age adjusted mortality rate, the death rate is currently higher among the vaccinated than the unvaccinated.  

The age adjusted mortality rates for vaccinated against unvaccinated for weeks 1 to 26 of 2021 are charted below. Overall, the chart shows that, over time, the weighted mortality rate for the vaccinated has steadily increased and by week 16 (23 April 2021), surpassed that for the unvaccinated. 

Week 1 ends 6 Jan 2021, Week 26 ends 2 July 2021

 

The chart suggests a normal seasonal mortality trend for the unvaccinated, with a winter peak on week 6, 12 February 2021, and a steady decline toward summer. In contrast, the pattern for the vaccinated is completely different. From week 24 onwards the mortality rates for the vaccinated and unvaccinated appear to be converging as summer begins. 

As the ONS data breaks down the data over time for the three categories of vaccinated (those within 21 days of first dose, those 21 days after first dose, and those after two doses), we can also plot mortality charts for each of these categories. The mortality rate, for week 26, up to 2 July, for the unvaccinated is around 25 deaths per 100,000. But there are big differences between the mortality rates for the different categories of vaccinated deaths. For example, for those after 21 days of first dose, the comparable mortality is around 89 deaths per 100,000 people (a number which has drastically increased since January), while for those vaccinated with two doses there were approximately 15 deaths per 100,000 in the same July period.

 


The trends for the different vaccination categories are also concerning. In contrast to the unvaccinated, the mortality rates for the vaccinated have initially increased from very low initial values, but then have increased, whilst that for the unvaccinated has decreased. The charts below show these patterns.

 


 


 

Since 19 March the double dose vaccination mortality rate has increased week-on-week more or less consistently. The mortality rate for those more than 21 days after first dose increased drastically in the spring (at week 14) and remained high thereafter. Mortality within 21 days of vaccination initially increased but looks to have stabilised, albeit with some noise. We will leave it to clinical colleagues to explain why there are such different patterns.

Because of the limitations and possible errors in the ONS data**, there are many caveats that need to be applied to our crude analysis (including some which are covered in the previous post). But we can conclude that the ONS's own data does not support the claims made for vaccine effectiveness/safety.  

It is also important to note that the population of vaccinated people is becoming sufficiently large and representative that the criticality of age adjustment becomes much diminished. We will be doing a follow-up analysis that takes account of this.


* For those who responded to this article saying they did not understand why we focus on all-cuase mortality:


 **Potential limitations and errors in the ONS data (with thanks to Clare Craig for identifying some of these)

  • Does not provide the raw age categorized data.
  • The age standardized score used by ONS relies on the 2011 census data to determine the population proportions in each age category. These proportions have changed since 2011 and, as we noted in this article, these differences can significantly change the results.
  • There are inconsistencies in vaccination numbers between the ONS data and the National Immunisation Management Service (NIMS) data.  For example, by week 26 NIMS has 28.1 million people over 18 who have had second does, but ONS has only 23.3 million. 
  • The ONS total population is 16.6 million short of the whole population. Only 12.6 million are under 18 so the remaining 4 million are omitted for some other reason.
  • The rates in the unvaccinated on 8th Jan are lower than the double vaccinated in summer. Also, on 8th January only 12% of over 65 year olds had been vaccinated, so the unvaccinated population should have had a death rate very similar to background levels. 
  • The wildly increasing weekly age adjusted mortality rates (for non-Covid related deaths) for the 38 million unvaccinated population in January are totally inconsistent with weekly changes in previous years. Although this population excludes the under 18s and the 1.2 million (mainly over 65s) who had by then recieved their first dose, we would not expect the mortality rate for this population to be drastically different to the mortality rate for England seen in recent years as reported in a different ONS report
  • Ultimately we need to exclude unnatual deaths such as murders, accidents and suicides since these may introduce bias between the cohorts, especially in the young age categories where the overall death numbers are small.

 Here is Table 4 data the raw data, for Covid-19 deaths, as provided by the ONS:


Here is Table 5 data the raw data, for all-cause deaths except for Covid-19, as provided by the ONS:


Finally, here is the data we used to calculate combined all-cause age adjusted mortality rates and the weighted vaccinated mortality rate.


***

The ONS definition of age-standardised mortality rates (click to enlarge)


Wednesday 15 September 2021

Paradoxes in the reporting of Covid19 vaccine effectiveness

The full pdf version of the following article (which includes the Appendix) can be found here.


Paradoxes in the reporting of Covid19 vaccine effectiveness
 

Why current studies (for or against vaccination) cannot be trusted and what we can do about it

 

Norman Fenton, Martin Neil and Scott McLachlan

 

Risk Information and Management Research

School of Electronic Engineering and Computer Science,

Queen Mary University of London

 

15 Sept 2021

 

The randomized controlled trials (RCTs) to establish the safety and effectiveness of Covid19 vaccines produced impressive results (Polack et al., 2020) but were inevitably limited in the way they assessed safety (Folegatti et al., 2020)[1] and are effectively continuing (Ledford, Cyranoski, & Van Noorden, 2020; Singh et al., 2021) . Ultimately, the safety and effectiveness of these vaccines will be determined by real world observational data over the coming months and years.

However, data from observational studies on vaccine effectiveness can easily be misinterpreted leading to incorrect conclusions. For example, we previously noted[2] the Public Health England data shown in Figure 1  for Covid19 cases and deaths of vaccinated and unvaccinated people up to June 2021. Overall, the death rate was three times higher in the vaccinated group, leading many to conclude that vaccination increases the risk of death from Covid19. But this conclusion was wrong for this data because, in each of the different age categories (under 50 and 50+), the death rate was lower in the vaccinated group.

Figure 1 Data from Public Health England, June 2021

This is an example of Simpson’s paradox (Pearl & Mackenzie, 2018). It arises here because most vaccinated people were in the 50+ category where most deaths occur. Specifically: a) a much higher proportion of those aged 50+ were vaccinated compared to those aged <50; and b) those aged 50+ are much more likely to die.

So, as shown in Figure 2(a), ‘age’ is a confounding variable. While it is reasonable to assume that death is dependent on age, in a proper RCT to determine the effectiveness of the vaccine we would need to break the dependency of vaccination on age as shown in Figure 2(b), by ensuring the same proportion of people were vaccinated in each age category.



Figure 2 Causal model reflecting the observed data

The Appendix demonstrates how this causal model, and Bayesian inference, can both explain the paradox and avoid it (by simulating an RCT). Using the model in Figure 2 (b), which avoids the confounding effect of age, we conclude (based only on the data in this study) that the (relative) risk of death is four times higher in the unvaccinated (0.417%) than the vaccinated (0.104%), meaning the absolute increase in risk of death is 0.313%  greater for the unvaccinated.

An excellent article by Jeffrey Morris[3] demonstrates the paradox in more detail using more recent data from Israel.

Clearly confounding factors like age (and also comorbidities) must, therefore, always be considered to avoid underestimating vaccine effectiveness data. However, the conclusions of these studies are also confounded by failing to consider non-Covid deaths, which will overestimate the safety of the vaccine if there were serious adverse reactions.

In fact, there are many other confounding factors that can compromise the results of any observational study into vaccine effectiveness (Krause et al., 2021). By ‘compromise’ we mean not just over- or under-estimate effectiveness, but - as in the example above - may completely reverse the results if we fail to adjust even for a single confounder (Fenton, Neil, & Constantinou, 2019).  

In particular, the following usually ignored confounding factors will certainly overestimate vaccine effectiveness. These include:

  • The classification of Covid19 deaths and hospitalizations. For those classified as Covid19 cases who die (whether due to Covid19 or some other condition), there is the issue of whether the patient is classified as dying ‘with’ Covid19 or ‘from’ Covid19. There may be differences between vaccinated and unvaccinated in the way this classification is made. The same applies to patients classified as Covid19 cases who are hospitalized.
  • The number of doses and amount of time since last dose used to classify whether a person has been vaccinated.  For example, any person testing positive for Covid19 or dying of any cause within 14 days of their second dose is now classified by the CDC as ‘unvaccinated’ (CDC, 2021). While this definition may make sense for determining effectiveness in preventing Covid19 infections, it  may drastically overestimate vaccine safety; this is because most serious adverse reactions from vaccines in general occur in the first 14 days (Scheifele, Bjornson, & Johnston, 1990; Stone, Rukasin, Beachkofsky, Phillips, & Phillips, 2019) and the same applies to Covid19 vaccines (Farinazzo et al., 2021; Mclachlan et al., 2021). There is also growing evidence that people hospitalized for any reason within 14 days of a vaccination are classified as unvaccinated and, for many, as Covid19 cases[4].
  • The accuracy of Covid19 testing and Covid19 case classification. These are critical factors since there may be different testing strategies for the unvaccinated compared to the vaccinated. For example, in the large observation study of the Pfizer vaccine effectiveness in Israel (Haas et al., 2021) unvaccinated asymptomatic people were much more likely to be tested than vaccinated asymptomatic people, resulting in the unvaccinated being more likely to be classified as Covid19 cases than vaccinated[5].

Even if we wish to simply study the effectiveness of the vaccine with respect to avoiding Covid infection (as opposed to avoiding death or hospitalization) there are many more factors that need to be considered than currently are.  To properly account for the interacting effects of all relevant factors that ultimately impact (or explain) observed data we need a causal model such as that in Figure 3.


Figure 3 Causal model to determine vaccine effectiveness

 

As in the simple model of Figure 2, the nodes in the model shown in Figure 3 correspond to relevant factors (some of which relate to individuals – like age, and some of which relate to the population – like whether lockdowns are in place) and an arc from one node to another means there is a direct causal/influential dependence in the direction of the arc. For example: younger people – and those who have immunity from previous Covid infection – are less likely to be vaccinated than older people; older people are more likely to have comorbidities and more likely to have symptoms if they are infected.  However, while those factors and relationships are widely considered in observational studies, most of the other factors in the model are not.

The first thing to note is that the model makes clear the critical distinction between whether a person is Covid19 infected (something which is not easily observable) and whether they are classified as a Covid19 case (i.e. the ones who are recorded as cases in any given study). The latter depends not just on whether they are genuinely infected but also on the accuracy of the testing and whether they are vaccinated. If (as in the Israel study described above) the unvaccinated are subject to more extensive (and potentially inaccurate) testing, then they are more likely to be erroneously classified as a case.  The model also makes clear the critical distinction between those who have been vaccinated (at least once) and those classified as vaccinated in the study. The latter depends on the number of doses, time since last dose, and whether the person tests positive. Moreover, whether a person gets more than one dose will depend on whether they suffered an adverse reaction first time; those who do and who do not get a second dose are generally classified as unvaccinated  - and this will compromise any studies of risk associated with the vaccine. Indeed, even the results of randomized controlled trials were compromised both by ‘removing’ those who died within 14 days of the second vaccination and ‘losing’ many subjects after the first dose[6].

The causal model makes clear that a person cannot become infected with the virus unless they come into contact with it. The latter depends not just on age, ethnicity and profession (so young people who live, work and travel in crowded environments are more likely to come into contact with the virus as are any people in a hospital environment) but also on changing population factors like lockdown restrictions in place and current population infection rate. Assuming a person comes into contact with the virus, whether they get infected depends on whether they have natural immunity and whether they are vaccinated.

If we had relevant data on all of the factors in the model then, as in the case of the simple model in the Appendix, we can capture the probabilistic dependence between each node and its immediate parents, and then use Bayesian inference to determine the true effect of vaccination. In principle, this enables us to properly explain all observed data, adjust for all confounding factors, and provide truly accurate measures of effectiveness. The problem is that several key variables are generally unobservable directly while many of the easily observable variables are simply not recorded. While we can incorporate expert judgment with observed statistical data to populate the model, this can be extremely complex and subjective. 

Moreover, if you think the model is already very complex, then it should be noted that it is far from fully comprehensive. Even before we consider all the additional factors and relationships needed to consider the outcomes of hospitalization and death (and the accuracy of reporting these), the model does not take account of: different treatments given; different morbidities and lifestyle choices; seasons over which data are collected; different strains of the virus; and many other factors.  Nor does it account for the fact that all observational data are biased (or ‘censored’) in the sense that it only contains information on people who are available for the study; so, for example, studies in particular countries will largely contain people of a specific ethnicity, while all studies will generally exclude certain classes of people (such as the homeless). This means that, while such studies could be useful in determining effectiveness at a ‘local’ level, their conclusions are not generalizable. Indeed, they may are completely unreliable because of another paradox (called collider or Berkson’s paradox) unless we have explicitly adjusted for this as described in (Fenton, 2020).

Given the impossibility of controlling for all these factors in randomized trials, and the overwhelming complexity of adjusting for them from observational data there is little we can reliably conclude from the data and studies so far. And we have not even mentioned the general failure of these studies to consider the impact and trade-offs of safety on effectiveness.

So, what can we do about this mess? We believe there is an extremely simple and objective solution: if we ignore the cost of vaccination, then ultimately we can all surely agree that the vaccine is effective overall if there are fewer deaths (from any cause) among the vaccinated than the unvaccinated. This combines both effectiveness and safety since it encapsulates the trade-off between them. It is not perfect, because there could be systemic differences in treatments given to vaccinated and unvaccinated[7], but it completely bypasses the problem of classifying Covid19 ‘cases’ which, as we have noted, compromises all studies so far.

So, provided that we can agree on an objective way to classify a person as vaccinated (and we propose that, for this purpose, the fairest way is to define anybody  as vaccinated if they have received at least one dose), then all we need to do is compare all-cause mortality rates in different age categories of the vaccinated v unvaccinated over a period of several months[8].  

A recent analysis does indeed look at all-cause deaths in vaccinated and unvaccinated  (Classen, 2021). The study shows that, for all three of the vaccines for which data were available, all-cause deaths is significantly higher in the vaccinated than the unvaccinated. However, this study did not account for age and hence its conclusions are also unreliable.  

We could immediately evaluate the effectiveness to date of vaccines in the UK by simply looking at the registered deaths since the start of the vaccination programme in December 2020. All we need to know for each registered death is the person’s age and whether they received at least one dose of the vaccine before death. Although a longer period would, of course, be better it is still sufficiently long to show a real effect if the vaccines work as claimed and if Covid19 is as deadly as claimed.

Moving forward we should certainly be collecting this simple data, but our concern is that (in many countries) the ‘control group’ (i.e. unvaccinated) may soon not be large enough for such a simple evaluation.

 

References

CDC. (2021). COVID-19 Breakthrough Case Investigations and Reporting | CDC. Retrieved September 15, 2021, from https://www.cdc.gov/vaccines/covid-19/health-departments/breakthrough-cases.html

Classen, B. (2021). US COVID-19 Vaccines Proven to Cause More Harm than Good Based on Pivotal Clinical Trial Data Analyzed Using the Proper Scientific Endpoint, “All Cause Severe Morbidity.” Trends in Internal Medicine, 1(1), 1–6. Retrieved from https://www.scivisionpub.com/pdfs/us-covid19-vaccines-proven-to-cause-more-harm-than-good-based-on-pivotal-clinical-trial-data-analyzed-using-the-proper-scientific--1811.pdf

Farinazzo, E., Ponis, G., Zelin, E., Errichetti, E., Stinco, G., Pinzani, C., … Zalaudek, I. (2021). Cutaneous adverse reactions after m‐RNA COVID‐19 vaccine: early reports from Northeast Italy. Journal of the European Academy of Dermatology and Venereology, 35(9), e548–e551. https://doi.org/10.1111/jdv.17343

Fenton, N. (2020). Why most studies into COVID19 risk factors may be producing flawed conclusions - and how to fix the problem. ArXiv. https://doi.org/http://arxiv.org/abs/2005.08608

Fenton, N. E., Neil, M., & Constantinou, A. (2019). Simpson’s Paradox and the implications for medical trials. Retrieved from http://arxiv.org/abs/1912.01422

Folegatti, P. M., Ewer, K. J., Aley, P. K., Angus, B., Becker, S., Belij-Rammerstorfer, S., … Oxford COVID Vaccine Trial Group. (2020). Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial. Lancet (London, England), 396(10249), 467–478. https://doi.org/10.1016/S0140-6736(20)31604-4

Haas, E. J., Angulo, F. J., McLaughlin, J. M., Anis, E., Singer, S. R., Khan, F., … Alroy-Preis, S. (2021). Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. Lancet (London, England), 397(10287), 1819–1829. https://doi.org/10.1016/S0140-6736(21)00947-8

Krause, P. R., Fleming, T. R., Peto, R., Longini, I. M., Figueroa, J. P., Sterne, J. A. C., … Henao-Restrepo, A.-M. (2021). Considerations in boosting COVID-19 vaccine immune responses. The Lancet, 0(0). https://doi.org/10.1016/S0140-6736(21)02046-8

Ledford, H., Cyranoski, D., & Van Noorden, R. (2020). The UK has approved a COVID vaccine — here’swhat scientists now want to know. Retrieved from https://www.nature.com/articles/d41586-020-03441-8?utm_source=Nature+Briefing&utm_campaign=597ee8dba8-briefing-dy-20201203&utm_medium

Mclachlan, S., Osman, M., Dube, K., Chiketero, P., Choi, Y., & Fenton, N. (2021). Analysis of COVID-19 vaccine death reports from the Vaccine Adverse Events Reporting System (VAERS) Database Interim: Results and Analysis. Retrieved from http://dx.doi.org/10.13140/RG.2.2.26987.26402

Pearl, J., & Mackenzie, D. (2018). The book of why : the new science of cause and effect. New York: Basic Books.

Polack, F. P., Thomas, S. J., Kitchin, N., Absalon, J., Gurtman, A., Lockhart, S., … C4591001 Clinical Trial Group. (2020). Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine. The New England Journal of Medicine, 383(27), 2603–2615. https://doi.org/10.1056/NEJMoa2034577

Scheifele, D. W., Bjornson, G., & Johnston, J. (1990). Evaluation of adverse events after influenza vaccination in hospital personnel. CMAJ : Canadian Medical Association Journal = Journal de l’Association Medicale Canadienne, 142(2), 127–130. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2295029

Singh, J. A., Kochhar, S., Wolff, J., Atuire, C., Bhan, A., Emanuel, E., … Upshur, R. E. G. (2021). Placebo use and unblinding in COVID-19 vaccine trials: recommendations of a WHO Expert Working Group. Nature Medicine, 27(4), 569–570. https://doi.org/10.1038/s41591-021-01299-5

Stone, C. A., Rukasin, C. R. F., Beachkofsky, T. M., Phillips, E. J., & Phillips, E. J. (2019). Immune‐mediated adverse reactions to vaccines. British Journal of Clinical Pharmacology, 85(12), 2694–2706. https://doi.org/10.1111/bcp.14112


 



[1] Some participants and sites were unblinded and non-randomised and others were effectively unblinded when they received paracetamol prior to jab

[2] https://probabilityandlaw.blogspot.com/2021/06/simpsons-paradox-in-interepretation.html

[3] https://www.covid-datascience.com/post/israeli-data-how-can-efficacy-vs-severe-disease-be-strong-when-60-of-hospitalized-are-vaccinated

[4] https://www.bitchute.com/video/lXrcpFe4V4U2/

[5] https://probabilityandlaw.blogspot.com/2021/05/important-caveats-to-pfizer-vaccine.html

[6] Some of the covid vax trials were unblinded, others were only single-blinded. Yet more were non-randomised and others were accidentally unblinded when the treatment recipients were given paracetamol prior to their covid jab

[7] There are multiple anecdotal reports that Australian hospitals are now giving ivermectin only to vaccinated patients

[8] https://probabilityandlaw.blogspot.com/2021/06/why-all-studies-so-far-into-risks-andor.html