Is vaccine efficacy a statistical illusion?

1 Dec 2021 upate: see update to this article showing that the same statistical illusion of efficacy occurs if newly vaccinated deaths are classified as unvaccinated.

To evaluate the risk/benefit of a vaccine for treating a virus, such as covid-19, we can compare the all-cause mortality rate of vaccinated against unvaccinated people on a periodic basis. If the mortality rate for those vaccinated is consistently lower than that for unvaccinated then we might conclude the vaccine must be beneficial.

Placebo Vaccination

Imagine that a placebo rather than a vaccine is quickly rolled out to a population of one million people of similar age and health. Let’s assume the weekly non-virus mortality rate for this population is 15 per 100,000 (100k), so we would expect about 150 out of the million to die in any given week. Because the placebo changes nothing, the mortality rates for both vaccinated and unvaccinated average the same 15 per 100k, each week every week. Hence, on average, what we should observe – as the ‘vaccination’ programme rolls out to most of the population - is shown in Table 1. Notice that the placebo vaccine roll-out programme is enacted at pace and the cumulative percentage of the population vaccinated rises to 98% within 12 weeks.

Table 1 Roll out of placebo ‘vaccine’. No observed differences in mortality rates (no population growth and each week total population reduced by previous week’s deaths)

			Vaccinated			Unvaccinated
Week	Population	Cumulative Percentage vaccinated	Deaths	Population	Mortality rate	Deaths	Population	Mortality rate
1	1,000,000	0.5	1	5,000	15	149	995,000	15
2	999,850	1	1	9,999	15	148	989,852	15
3	999,700	2	3	19,994	15	147	979,706	15
4	999,550	4	6	39,982	15	144	959,568	15
5	999,400	7	10	69,958	15	139	929,442	15
6	999,250	14	21	139,895	15	129	859,355	15
7	999,100	28	42	279,748	15	108	719,352	15
8	998,950	45	67	449,528	15	82	549,423	15
9	998,801	65	97	649,220	15	52	349,580	15
10	998,651	80	120	798,921	15	30	199,730	15
11	998,501	93	139	928,606	15	10	69,895	15
12	998,351	98	147	978,384	15	3	19,967	15
13	998,201	98.5	147	983,228	15	2	14,973	15
14	998,052	98.6	148	984,079	15	2	13,973	15
15	997,902	98.7	148	984,929	15	2	12,973	15
16	997,752	98.9	148	986,777	15	2	10,975	15
17	997,603	99	148	987,627	15	1	9,976	15
18	997,453	99.1	148	988,476	15	1	8,977	15
19	997,303	99.2	148	989,325	15	1	7,978	15
20	997,154	99.3	149	990,174	15	1	6,980	15

 

Now suppose there is a one-week delay in the reporting of deaths. Such delays are routine in statistical reporting of mortality and vaccine data. Then the data reported by the authorities is different from reality, here shown in Table 2, which is the same as Table 1 but where the death totals are simply ‘shifted’ down one week.

Table 2 Death reporting delayed by one week

			Vaccinated			Unvaccinated
Week	Population	Cumulative Percentage vaccinated	Deaths	Population	Mortality rate	Deaths	Population	Mortality rate
1	1,000,000	0.5	-	5,000	-	-	995,000	-
2	999,850	1	1	9,999	7.50	149	989,852	15.08
3	999,700	2	1	19,994	7.50	148	979,706	15.16
4	999,550	4	3	39,982	7.50	147	959,568	15.31
5	999,400	7	6	69,958	8.57	144	929,442	15.49
6	999,250	14	10	139,895	7.50	139	859,355	16.22
7	999,100	28	21	279,748	7.50	129	719,352	17.92
8	998,950	45	42	449,528	9.33	108	549,423	19.64
9	998,801	65	67	649,220	10.39	82	349,580	23.57
10	998,651	80	97	798,921	12.19	52	199,730	26.25
11	998,501	93	120	928,606	12.91	30	69,895	42.86
12	998,351	98	139	978,384	14.24	10	19,967	52.51
13	998,201	98.5	147	983,228	14.93	3	14,973	20.00
14	998,052	98.6	147	984,079	14.99	2	13,973	16.07
15	997,902	98.7	148	984,929	14.99	2	12,973	16.16
16	997,752	98.9	148	986,777	14.97	2	10,975	17.73
17	997,603	99	148	987,627	14.99	2	9,976	16.50
18	997,453	99.1	148	988,476	14.99	1	8,977	16.67
19	997,303	99.2	148	989,325	14.99	1	7,978	16.88
20	997,154	99.3	148	990,174	14.99	1	6,980	17.15

 (Update) Here's a 60-second video showing how to this is done in Excel and proving there are no tricks involved other than simply shifting the deaths down by one week:

Suppose we want to examine and compare the mortality rates of the unvaccinated and vaccinated cohorts based on the data in Table 2. Figure 1 shows this comparison, and we can see that the mortality rate is consistently lower for the vaccinated than that for the unvaccinated throughout the roll out of the vaccination programme and it reduces as soon as vaccination nears population saturation at close to 100%.

 

Figure 1 Reported weekly mortality rates vaccinated against unvaccinated

We might conclude that those who remain unvaccinated look to be suffering much higher levels of mortality than the vaccinated. The reporting delay therefore creates a completely artificial impression that the vaccine must be highly effective.  In fact, it looks like a magic ‘cure all’ wonder drug!

The fact that the mortality rate of the unvaccinated peaks when the percentage of those vaccinated peaks should ring some alarm bells that something strange is going on (unless there is independent evidence that the virus was peaking at the same time).

 

ONS data on Covid-19 Vaccination

While the placebo vaccine example was purely hypothetical, Figure 2 shows the vaccinated against unvaccinated mortality using the data in the latest ONS report mortality in England by Covid-19 vaccination status (weeks 1 to 38)[1], complemented by NIMS vaccination survey data (up to week 27 only). Here we show other-than covid mortality to remove the virus signal.

 

Figure 2 Reported weekly other-than covid mortality rates for vaccinated versus unvaccinated for 60-69 age group for weeks 1-38 2021

Note that we see the same features as the shifted graph in Figure 1. In other words, a perfectly reasonable explanation for what is observed here could be that there is no difference in mortality rates between vaccinated and unvaccinated and the mortality differences are simply a result of a delay in death reporting. Moreover, given we have removed covid deaths (which were only a small percentage of all-cause deaths in the reported data) we get a near identical result for non-covid mortality to that which would result if the vaccine were a placebo! Thus, we appear to have created a statistical illusion of vaccine efficacy.

It is important to note that we are not claiming the death reporting is delayed in the ONS data. In fact our work on this is ongoing and we believe that the most plausible explanation is misclassification and underestimation of proportion of unvaccinated. 

If this is not a statistical illusion how is it possible that the unvaccinated are dying from non-covid causes at a higher rate than vaccinated? Also how is it possible that, at the time vaccination rates are ramped up to nearly 100% of the population, the nonvaccinated are dying from non-covid deaths at almost twice the rate of those who are vaccinated?

These same patterns are also observable in the 70-79 and 80+ age groups (with the mortality peaks for the unvaccinated appearing at different weeks because these age groups received vaccinations earlier). This strongly suggests that what we are observing is a genuine statistical illusion unexplainable by any real impact of the vaccine on mortality rates. 

 

Consider a deadly placebo

It is also important to note that even if the actual mortality rate for the vaccinated was higher than that of the unvaccinated, where the vaccine was causing death, as a side effect, we would still likely observe the same illusion.

To see this effect let’s revisit our placebo vaccine example and make a small change to Table 1 where instead of a mortality rate of 15 per 100k for the vaccinated, suppose it is 17 per 100k (a rise in mortality of approximately 13%). So, the placebo vaccine is killing two more people per 100k and gives no mortality benefit otherwise. In this scenario the reported mortality rate for the ‘deadly placebo’ is compared to the first ‘placebo’ scenario, in Figure 3. Even here we see the illusion that the mortality rate for the vaccinated is lower than unvaccinated. Both scenarios are the opposite of reality, and both look interchangeable. This means the chance of picking up a vaccine side effect signal is close to impossible and instead the illusion is created of vaccine efficacy.

Figure 3 Reported weekly mortality rates vaccinated against unvaccinated for ‘placebo’ scenario and ‘deadly placebo’ scenarios

 

The illusion of declining vaccine efficacy

Finally, it is important to note that the same statistical illusion applies to all measures of vaccine efficacy whether they be cases, hospitalizations, or deaths. In fact, replacing the number of deaths in Table 1 with number of cases, with a one week reporting delay, would result in vaccine efficacy rates as shown in Figure 4.

Figure 4 Reported vaccine efficacy rates equivalent for placebo vaccine

This occurs when the actual placebo vaccine efficacy for cases is zero.

This reporting bias is one type of bias that might be called ‘reporting lag censoring’, a phenomenon whereby structural or process factors systematically interfere with when data is handled and reported with the consequential effect that it is then misinterpreted, leading to false conclusions.

 

UPDATE: Here is a 25 minute lecture explain the background and ongoing work:

 

---

[1] https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/deathsinvolvingcovid19byvaccinationstatusengland/deathsoccurringbetween2januaryand24september2021