## Sunday 14 November 2021

### Is vaccine efficacy a statistical illusion?

1 Dec 2021 upate: see update to this article showing that the same statistical illusion of efficacy occurs if newly vaccinated deaths are classified as unvaccinated.

To evaluate the risk/benefit of a vaccine for treating a virus, such as covid-19, we can compare the all-cause mortality rate of vaccinated against unvaccinated people on a periodic basis. If the mortality rate for those vaccinated is consistently lower than that for unvaccinated then we might conclude the vaccine must be beneficial.

Placebo Vaccination

Imagine that a placebo rather than a vaccine is quickly rolled out to a population of one million people of similar age and health. Let’s assume the weekly non-virus mortality rate for this population is 15 per 100,000 (100k), so we would expect about 150 out of the million to die in any given week. Because the placebo changes nothing, the mortality rates for both vaccinated and unvaccinated average the same 15 per 100k, each week every week. Hence, on average, what we should observe – as the ‘vaccination’ programme rolls out to most of the population - is shown in Table 1. Notice that the placebo vaccine roll-out programme is enacted at pace and the cumulative percentage of the population vaccinated rises to 98% within 12 weeks.

Table 1 Roll out of placebo ‘vaccine’. No observed differences in mortality rates (no population growth and each week total population reduced by previous week’s deaths)

 Vaccinated Unvaccinated Week Population Cumulative Percentage vaccinated Deaths Population Mortality rate Deaths Population Mortality rate 1 1,000,000 0.5 1 5,000 15 149 995,000 15 2 999,850 1 1 9,999 15 148 989,852 15 3 999,700 2 3 19,994 15 147 979,706 15 4 999,550 4 6 39,982 15 144 959,568 15 5 999,400 7 10 69,958 15 139 929,442 15 6 999,250 14 21 139,895 15 129 859,355 15 7 999,100 28 42 279,748 15 108 719,352 15 8 998,950 45 67 449,528 15 82 549,423 15 9 998,801 65 97 649,220 15 52 349,580 15 10 998,651 80 120 798,921 15 30 199,730 15 11 998,501 93 139 928,606 15 10 69,895 15 12 998,351 98 147 978,384 15 3 19,967 15 13 998,201 98.5 147 983,228 15 2 14,973 15 14 998,052 98.6 148 984,079 15 2 13,973 15 15 997,902 98.7 148 984,929 15 2 12,973 15 16 997,752 98.9 148 986,777 15 2 10,975 15 17 997,603 99 148 987,627 15 1 9,976 15 18 997,453 99.1 148 988,476 15 1 8,977 15 19 997,303 99.2 148 989,325 15 1 7,978 15 20 997,154 99.3 149 990,174 15 1 6,980 15

Now suppose there is a one-week delay in the reporting of deaths. Such delays are routine in statistical reporting of mortality and vaccine data. Then the data reported by the authorities is different from reality, here shown in Table 2, which is the same as Table 1 but where the death totals are simply ‘shifted’ down one week.

Table 2 Death reporting delayed by one week

 Vaccinated Unvaccinated Week Population Cumulative Percentage vaccinated Deaths Population Mortality rate Deaths Population Mortality rate 1 1,000,000 0.5 - 5,000 - - 995,000 - 2 999,850 1 1 9,999 7.50 149 989,852 15.08 3 999,700 2 1 19,994 7.50 148 979,706 15.16 4 999,550 4 3 39,982 7.50 147 959,568 15.31 5 999,400 7 6 69,958 8.57 144 929,442 15.49 6 999,250 14 10 139,895 7.50 139 859,355 16.22 7 999,100 28 21 279,748 7.50 129 719,352 17.92 8 998,950 45 42 449,528 9.33 108 549,423 19.64 9 998,801 65 67 649,220 10.39 82 349,580 23.57 10 998,651 80 97 798,921 12.19 52 199,730 26.25 11 998,501 93 120 928,606 12.91 30 69,895 42.86 12 998,351 98 139 978,384 14.24 10 19,967 52.51 13 998,201 98.5 147 983,228 14.93 3 14,973 20.00 14 998,052 98.6 147 984,079 14.99 2 13,973 16.07 15 997,902 98.7 148 984,929 14.99 2 12,973 16.16 16 997,752 98.9 148 986,777 14.97 2 10,975 17.73 17 997,603 99 148 987,627 14.99 2 9,976 16.50 18 997,453 99.1 148 988,476 14.99 1 8,977 16.67 19 997,303 99.2 148 989,325 14.99 1 7,978 16.88 20 997,154 99.3 148 990,174 14.99 1 6,980 17.15

(Update) Here's a 60-second video showing how to this is done in Excel and proving there are no tricks involved other than simply shifting the deaths down by one week:

Suppose we want to examine and compare the mortality rates of the unvaccinated and vaccinated cohorts based on the data in Table 2. Figure 1 shows this comparison, and we can see that the mortality rate is consistently lower for the vaccinated than that for the unvaccinated throughout the roll out of the vaccination programme and it reduces as soon as vaccination nears population saturation at close to 100%.

Figure 1 Reported weekly mortality rates vaccinated against unvaccinated

We might conclude that those who remain unvaccinated look to be suffering much higher levels of mortality than the vaccinated. The reporting delay therefore creates a completely artificial impression that the vaccine must be highly effective.  In fact, it looks like a magic ‘cure all’ wonder drug!

The fact that the mortality rate of the unvaccinated peaks when the percentage of those vaccinated peaks should ring some alarm bells that something strange is going on (unless there is independent evidence that the virus was peaking at the same time).

ONS data on Covid-19 Vaccination

While the placebo vaccine example was purely hypothetical, Figure 2 shows the vaccinated against unvaccinated mortality using the data in the latest ONS report mortality in England by Covid-19 vaccination status (weeks 1 to 38)[1], complemented by NIMS vaccination survey data (up to week 27 only). Here we show other-than covid mortality to remove the virus signal.

Figure 2 Reported weekly other-than covid mortality rates for vaccinated versus unvaccinated for 60-69 age group for weeks 1-38 2021

Note that we see the same features as the shifted graph in Figure 1. In other words, a perfectly reasonable explanation for what is observed here could be that there is no difference in mortality rates between vaccinated and unvaccinated and the mortality differences are simply a result of a delay in death reporting. Moreover, given we have removed covid deaths (which were only a small percentage of all-cause deaths in the reported data) we get a near identical result for non-covid mortality to that which would result if the vaccine were a placebo! Thus, we appear to have created a statistical illusion of vaccine efficacy.

It is important to note that we are not claiming the death reporting is delayed in the ONS data. In fact our work on this is ongoing and we believe that the most plausible explanation is misclassification and underestimation of proportion of unvaccinated.

If this is not a statistical illusion how is it possible that the unvaccinated are dying from non-covid causes at a higher rate than vaccinated? Also how is it possible that, at the time vaccination rates are ramped up to nearly 100% of the population, the nonvaccinated are dying from non-covid deaths at almost twice the rate of those who are vaccinated?

These same patterns are also observable in the 70-79 and 80+ age groups (with the mortality peaks for the unvaccinated appearing at different weeks because these age groups received vaccinations earlier). This strongly suggests that what we are observing is a genuine statistical illusion unexplainable by any real impact of the vaccine on mortality rates.

It is also important to note that even if the actual mortality rate for the vaccinated was higher than that of the unvaccinated, where the vaccine was causing death, as a side effect, we would still likely observe the same illusion.

To see this effect let’s revisit our placebo vaccine example and make a small change to Table 1 where instead of a mortality rate of 15 per 100k for the vaccinated, suppose it is 17 per 100k (a rise in mortality of approximately 13%). So, the placebo vaccine is killing two more people per 100k and gives no mortality benefit otherwise. In this scenario the reported mortality rate for the ‘deadly placebo’ is compared to the first ‘placebo’ scenario, in Figure 3. Even here we see the illusion that the mortality rate for the vaccinated is lower than unvaccinated. Both scenarios are the opposite of reality, and both look interchangeable. This means the chance of picking up a vaccine side effect signal is close to impossible and instead the illusion is created of vaccine efficacy.

Figure 3 Reported weekly mortality rates vaccinated against unvaccinated for ‘placebo’ scenario and ‘deadly placebo’ scenarios

The illusion of declining vaccine efficacy

Finally, it is important to note that the same statistical illusion applies to all measures of vaccine efficacy whether they be cases, hospitalizations, or deaths. In fact, replacing the number of deaths in Table 1 with number of cases, with a one week reporting delay, would result in vaccine efficacy rates as shown in Figure 4.

Figure 4 Reported vaccine efficacy rates equivalent for placebo vaccine

This occurs when the actual placebo vaccine efficacy for cases is zero.

This reporting bias is one type of bias that might be called ‘reporting lag censoring’, a phenomenon whereby structural or process factors systematically interfere with when data is handled and reported with the consequential effect that it is then misinterpreted, leading to false conclusions.

UPDATE: Here is a 25 minute lecture explain the background and ongoing work:

[1] https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/deathsinvolvingcovid19byvaccinationstatusengland/deathsoccurringbetween2januaryand24september2021