Is the Pfizer vaccine as effective as claimed?

17 August 2021 Update: After sending a letter to The Lancet in May 2021 summarising our concerns described here about the study, the editors (when prompted 3 weeks later) said they were waiting for the authors to respond before they could publish the letter. Four months on we have still heard nothing back from The Lancet. We have therefore now also added the letter below. It is worth noting that, since we wrote this article, several studies from Israel have shown that the effectiveness of the vaccine is indeed much lower than 95%. Also see this post about our general concerns on all studies on vaccine effectiveness.

There was massive media fanfare over the study (published in The Lancet) in Israel on the effectiveness of the Pfizer vaccine.

Notwithstanding the fact that 8 of the 15 authors "hold stock and share options in Pfizer"* the results look genuinely impressive and provide support for the hypothesis that the vaccine is effective in preventing infection. In particular, the raw data (Table 2 of the paper**) states the following

• Between 24 Jan 2021 and 3 April 2021 there were 109,876 'cases' of SARS-Cov-2 found among those unvaccinated*** compared to just 6,266 'cases' found among those vaccinated.
• The table also provides the 'incident rate per 100,000 person days' which is: 91.5 for unvaccinated compared to 3.1 for vaccinated 
• Based on these data the (adjusted) 'vaccine effectiveness' measure**** is calculated as 95.3% (hence the headline figure picked up by all main stream media).

There are, however, issues with the study and its analysis which mean the 95% effectiveness measure is exaggerated. In this article Will Jones argues that the researchers have not adjusted for the declining infection rate during the study period and that when you do so, the effectiveness drops to 74% (in the over 65's). 

A different problem with the study (that we focus on here) aises from the statement found on page 8 of the paper:

What this is saying is that, whereas unvaccinated people continued to be regularly and routinely subject to PCR tests, vaccinated people no longer had to be. The number of 'cases' stated in Table 2 is, of course, simply the number of positive PCR test outcomes (which includes false positives). If you stop testing vaccinated people then you are not going to find any 'cases' among them. The paper says that 19% of the tests were, however, on 'exempted', i.e. vaccinated people. But, this still means unvaccinated people were much more likely to be tested than vaccinated people, so we have to take account of the absolute number of tests performed on both vaccinated and unvaccinated.

We know that there were 4.4 million PCR tests and that 19% of these were on vaccinated people.  Hence, we can conlude that there were:

• 836,000 tests on vaccinated people (of whom there were 4,714,932, making up 72.1% of the population; so on average approximately one in six vaccinated people received a PCR test);
  
• 3,564,000 tests on unvaccinated people (of whom there were 1,823,979;  so, on average, each unvaccinated person received two PCR tests)

So, the number of 'cases' per 1000 tests were:

• 30.8 for unvaccinated people  (109,876 divided by 3,564,000 times 1000)
  
• 7.5 for vaccinated people  (6,266 divided by 836,000 times 1000)

Using the simple 'cases per 1000 tests' (rather than the biased 'incident rate per 100,000 person days'),   results in an approximate 'vaccine effectiveness' measure of 75.7%. While this is much less than the 95% headline figure, it is still impressive, so it is strange why the study failed to account for the difference in proportions tested. 

It appears that the failure to adjust the vaccine effectiveness calculation for different testing protocols for vaccinated and unvaccinated people is not restricted to this Pfizer study in Israel. The data in the FDA briefing document on the Pfizer vaccine (dated 10 Dec 2020) suggests there was a similar problem with the phase 3 trial of the vaccine. This was a randomized, double-blinded and placebo-controlled trial of the vaccine in 44,000 uninfected participants. It similarly reports a 95% effectiveness measure based on the fact that (post injection) there were 162 confirmed Covid-19 cases among the placebo participants compared to just 8 among the vaccinated participants.  However, the study also reports that there were a much larger number of 'suspected but unconfirmed' cases and that these were more evenly spread between placebo participants (1,816 such cases) and vaccinated participants (1,594 such cases). This seems to suggest that a disproportionately small number of vaccinated participants with symptoms received PCR tests compared to placebo participants with symptoms.

Clearly the failure to properly adjust for both a decreasing infection rate and different testing protocols for vaccinated and unvaccinated people casts doubt on the validity of the studies. 

It is also worth noting that, even if we ignore all of the above issues and accept as undisputed the number for 'COVID-19 related deaths' in the Israel study (715 among the unvaccinated and 138 among the vaccinated), then the absolute percentage increase in risk of death for an unvaccinated person is just 0.036%. That means that, even if we accept the 95% effectiveness measure, for every 10,000 unvaccinated people, about 3 or 4 would die as a result of not being vaccinated. And this brings us to the final (and critical) problem with the study. It does not provide any information about the number of adverse reactions - in particular the number of deaths - due to the vaccine. Hence, it does not provides the necessary information to make an informed decision about the overall risk/benefit of the vaccine.

We submitted the following 250-word response to The Lancet over a week ago summarising the above concerns about the article, but the response is still "With the Editor".

The article [1] provides impressive support for the effectiveness of the Pfizer vaccine, but important limitations in the overall analysis mean the results over-estimate vaccine effectiveness. One is that the study does not adjust for declining infection rate [2]. But there is also failure to properly adjust for the different testing protocols for vaccinated and unvaccinated people (page 8 of the paper). Whereas unvaccinated people continued to be regularly and routinely subject to PCR tests, vaccinated people no longer had to be. Although 836,000 (19%) of the 4.4 million PCR tests were on vaccinated people, 3,564,000 were on unvaccinated.

So, using the Table 2 data (109,876 cases among unvaccinated, and 6,266 among vaccinated) the number of 'cases' per 1000 tests were: 

• 30.8 for unvaccinated 
• 7.5 for vaccinated

Using 'cases per 1000 tests' results in an approximate 'vaccine effectiveness' measure of 75.7%, compared to the 95% headline figure. 

Failure to account for different testing protocols is also evident in the phase 3 trial of the Pfizer vaccine [3]. It similarly reports a 95% effectiveness measure based on 162 confirmed cases among placebo participants compared to just 8 among vaccinated participants. However, the study also reports a much larger number of 'suspected but unconfirmed' cases more evenly spread between placebo participants (1,816) and vaccinated participants (1,594). This suggests a disproportionately small number of vaccinated participants with symptoms received PCR tests compared to placebo participants with symptoms.

[1] Haas et al: “Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data” https://doi.org/10.1016/S0140-6736(21)00947-8

[2] Jones: “Study Claims Pfizer Vaccine is 95% Effective in Over 65s. But Should That Be 74%?” https://lockdownsceptics.org/2021/05/07/study-claims-pfizer-vaccine-is-95-effective-in-over-65s-but-should-that-be-74/

[3] FDA Briefing DocumentPfizer-BioNTech COVID-19 Vaccine, https://www.fda.gov/media/144245/download

 

*screenshot of declared interests in the paper:

**Table 2 screenshot from the paper:

***Although Table 2 states that there were a total of 109,876 'cases' among the unvaccinated, there seems to be an error in the table in that the total number of asymptomatic cases (49,138) and symptomatic cases (39,065) do not sum to 109,876

****The 'vaccine effectiveness' measure is defined as: 100 times (1 -  the incident rate ratio). The incident rate ratio is (approximately) the incident rate of vaccinated divided by the incident rate of unvaccinated.

 

Postscript: The study provides interesting insights into the separate issue of 'asymptomatic' infection that we have covered extensively on this blog. For example, Table 2 shows that, among the unvaccinated, there were 49,138 asymptomatic 'cases' compared to 39,065 symptomatic 'cases', i.e. 56% of all those testing positive (and classified as a 'case') were asymptomatic. It is likely that most of the positives among the asymptomatics were false positives. This is because, especially at times when the infection rate is low, a false positive PCR test rate of, say just 0.4%, would still mean that the majority of positive tests among asymptomatics are false. See  here and here.