There was massive media fanfare over the study (published in The Lancet) in Israel on the effectiveness of the Pfizer vaccine.
Notwithstanding the fact that 8 of the 15 authors "hold stock and share options in Pfizer"* the results look genuinely impressive and provide support for the hypothesis that the vaccine is effective in preventing infection. In particular, the raw data (Table 2 of the paper**) states the following
- Between 24 Jan 2021 and 3 April 2021 there were 109,876 'cases' of SARS-Cov-2 found among those unvaccinated*** compared to just 6,266 'cases' found among those vaccinated.
- The table also provides the 'incident rate per 100,000 person days' which is: 91.5 for unvaccinated compared to 3.1 for vaccinated
- Based on these data the (adjusted) 'vaccine effectiveness' measure**** is calculated as 95.3% (hence the headline figure picked up by all main stream media).
There are, however, issues with the study and its analysis which mean the 95% effectiveness measure is exaggerated. In this article Will Jones argues that the researchers have not adjusted for the declining infection rate during the study period and that when you do so, the effectiveness drops to 74% (in the over 65's).
A different problem with the study (that we focus on here) aises from the statement found on page 8 of the paper:
What this is saying is that, whereas unvaccinated people continued to be regularly and routinely subject to PCR tests, vaccinated people no longer had to be. The number of 'cases' stated in Table 2 is, of course, simply the number of positive PCR test outcomes (which includes false positives). If you stop testing vaccinated people then you are not going to find any 'cases' among them. The paper says that 19% of the tests were, however, on 'exempted', i.e. vaccinated people. But, this still means unvaccinated people were much more likely to be tested than vaccinated people, so we have to take account of the absolute number of tests performed on both vaccinated and unvaccinated.
We know that there were 4.4 million PCR tests and that 19% of these were on vaccinated people. Hence, we can conlude that there were:
- 836,000 tests on vaccinated people (of whom there were 4,714,932, making up 72.1% of the population; so on average approximately one in six vaccinated people received a PCR test);
- 3,564,000 tests on unvaccinated people (of whom there were 1,823,979; so, on average, each unvaccinated person received two PCR tests)
So, the number of 'cases' per 1000 tests were:
- 30.8 for unvaccinated people (109,876 divided by 3,564,000 times 1000)
- 7.5 for vaccinated people (6,266 divided by 836,000 times 1000)
Using the simple 'cases per 1000 tests' (rather than the biased 'incident rate per 100,000 person days'), results in an approximate 'vaccine effectiveness' measure of 75.7%. While this is much less than the 95% headline figure, it is still impressive, so it is strange why the study failed to account for the difference in proportions tested.
It appears that the failure to adjust the vaccine effectiveness calculation for different testing protocols for vaccinated and unvaccinated people is not restricted to this Pfizer study in Israel. The data in the FDA briefing document on the Pfizer vaccine (dated 10 Dec 2020) suggests there was a similar problem with the phase 3 trial of the vaccine. This was a randomized, double-blinded and placebo-controlled
trial of the vaccine in 44,000 uninfected participants. It similarly reports a 95% effectiveness measure based on the fact that (post injection) there were 162 confirmed Covid-19 cases among the placebo participants compared to just 8 among the vaccinated participants. However, the study also reports that there were a much larger number of 'suspected but unconfirmed' cases and that these were more evenly spread between placebo participants (1,816 such cases) and vaccinated participants (1,594 such cases). This seems to suggest that a disproportionately small number of vaccinated participants with symptoms received PCR tests compared to placebo participants with symptoms.
Clearly the failure to properly adjust for both a decreasing infection rate and different testing protocols for vaccinated and unvaccinated people casts doubt on the validity of the studies.
It is also worth noting that, even if we ignore all of the above issues and accept as undisputed the number for 'COVID-19 related deaths' in the Israel study (715 among the unvaccinated and 138 among the vaccinated), then the absolute percentage increase in risk of death for an unvaccinated person is just 0.036%. That means that, even if we accept the 95% effectiveness measure, for every 10,000 unvaccinated people, about 3 or 4 would die as a result of not being vaccinated. And this brings us to the final (and critical) problem with the study. It does not provide any information about the number of adverse reactions - in particular the number of deaths - due to the vaccine. Hence, it does not provides the necessary information to make an informed decision about the overall risk/benefit of the vaccine.
We submitted a 250-word response to The Lancet over a week ago
summarising the above concerns about the article, but the response is
still "With the Editor".
*screenshot of declared interests in the paper:
**Table 2 screenshot from the paper:
***Although Table 2 states that there were a total of 109,876 'cases' among the unvaccinated, there seems to be an error in the table in that the total number of asymptomatic cases (49,138) and symptomatic cases (39,065) do not sum to 109,876
****The 'vaccine effectiveness' measure is defined as: 100 times (1 - the incident rate ratio). The incident rate ratio is (approximately) the incident rate of vaccinated divided by the incident rate of unvaccinated.
Postscript: The study provides interesting insights into the separate issue of 'asymptomatic' infection that we have covered extensively on this blog. For example, Table 2 shows that, among the unvaccinated, there were 49,138
asymptomatic 'cases' compared to 39,065 symptomatic 'cases', i.e. 56%
of all those testing positive (and classified as a 'case') were
asymptomatic. It is likely that most of the positives among the
asymptomatics were false positives. This is because, especially at times when the infection rate is low, a false positive PCR test rate of, say just 0.4%, would still mean that the majority of positive tests among asymptomatics are false. See here and here.
