Thursday, 30 June 2022

Response to Susan Oliver video “Antivaxxers fooled by p-hacking and apples to oranges comparison”

 

The video and the tweet publicising it

On 26 June 2022 Susan Oliver published a video on YouTube titled “Antivaxxers fooled by p-hacking and apples to oranges comparison” in response to a preprint [1] by 8 authors, one of whom was well-know BMJ Senior Editor Peter Doshi. She refers to the paper as the “Doshi paper” and we will use the same reference here even though Doshi is the last, rather than first, named author.  The paper demonstrates the increased risk of serious adverse events (SAEs) arising from the Pfizer and Moderna covid vaccine trials. Susan summarised her view of the paper in this tweet (which included the link to the video) that was retweeted by people like Prof Sir David Spiegelhalter (a world renowned expert on probability and risk) and Prof Peter Hansen (Econometrician, Data Scientist, and Latene Distinguished Professor of Economics at UNC, Chapel Hill):



 

What Susan says in the video and why it totally misrepresents the Doshi paper

Susan spends 3 minutes highlighting a number of people she refers to as “anti-vaxxers” who tweeted about the paper, including Jordan Peterson who she refers to as a "self-declared best-selling author" (note: his 2018 book sold over 3 million copies and was number 1 on Amazon).  Susan then states:

 “It’s basically just a rubbish paper that uses a technique known as p-hacking followed by some apples stuff oranges comparisons”.

Interestingly, despite the video title, Susan spends less than 30 seconds describing what p-hacking is and instead refers to a paper about it [2] (we agree entirely with the general concerns raised about p-hacking and show how it is avoided using Bayesian hypothesis testing [3]). But the key flaw in Susan's criticism is that the “Doshi paper” is not an example of p-hacking at all. They do not use p-values and, also contrary to the continued assertions of Susan, they make no claims at all of statistical significance. Rather, the paper provides risk differences and risk ratios with 95% confidence intervals (CIs) for the various different comparisons of vaccine v placebo. For example, here is their table of results for all serious adverse events (SAEs) and also of the subset of serious adverse events of special interest (serious AESIs):


If the authors had been “p-hacking” they would have chosen a p-value like 0.05 and would have added, for each comparison of vaccine v placebo, a ‘significance statistic’ and arrived at at least one example where the statistic was less than 0.05.  Then they would claim, for example, that the increased SAE rate was ‘significant’.  They do nothing like that at all.

Susan then claims that only by ‘combining’ the data from the different trials does Doshi get the (mythically claimed) ‘significant results’ and that such combining should simply not be done (this is one of her ‘apples and oranges’ comparison argument). But, while it is true that the paper does also look at the combined numbers for each class of SAE, it turns out that in each case, the risk ratio for the combined numbers is actually less than for the Pfizer trial alone. For example, for all SAEs the (median) risk ratio for Pfizer v placebo is 1.36 compared to just 1.15 for combined v placebo: the results are less, not more, ‘significant’. Our own Bayesian analysis of the results presented below makes this very clear.

Susan’s final criticisms of the Doshi paper concerns the selection of SAEs and the possibility of ‘double counting’. Regarding selection, the events included and not included are governed by the WHO endorsed Brighton scheme, and are not decided by the authors, so this is a critical error Susan makes. The Brighton list was created a priori, based on data before the any results were released from the trials. Any double counting, such as with the diarrhoea and abdominal pain example she uses, are a direct effect of the fact that the data are not public. There’s merit to both measures - counting number of participants (with any SAE) and number of events. If one person has two SAEs that is worse than one person having one SAE. “Double counting” sounds bad, but this is not double counting. Doshi et al are measuring how many SAEs occur in the vaccine group versus the placebo group. If Diarrhoea and abdominal pain were each recorded as a SAE, then that is two SAEs. We don’t know which ones were in the same person as Pfizer/Moderna have not released IPD. In any case, the authors recognise the issue that, because some SAEs occur in the same person, the SAEs are not all independent events; they note it in the paper, and introduce an adjustment to standard error to account for it. It is unclear whether the adjustment is sufficient, but it actually weakens their case (it increases the size of the confidence intervals) - so they can hardly be accused of bias. 

Further regarding double counting, SAEs are counted individually to avoid them being hidden. So, if you get renal failure and then your penis drops off that should be two SAEs, not one.  One person having three SAEs (renal failure, penis drops off, stroke) could be considered as serious as three people having a stroke; so, although some clinicians disagree, it is entirely reasonable to count SAEs separately.  But Susan does not appear to understand what a SAE is. She assumes something like diarrhoea cannot be a SAE because lots of diarrhoea happens to be mild. But most covid is not serious, either. So diarrhoea can be a SAE if it’s serious enough and meets the regulatory criteria. And it’s a leading cause of death in some places.

That addresses all the ‘flaws’ that Susan claims about the paper. It is also important to note that, even when all the SAEs in the Pfizer and Moderna trial are combined, the absolute risk increase is fairly small -  a fact already made clear by Doshi et al. (although this is to be balanced against the very low risks of severe covid, which is in essence the core message of the paper). They state that, in this case, the absolute risk increase (95% CI) is between 2.1 to 22.9 events per 10,000 participants.  In our Bayesian analysis the median absolute risk increase is 12.9 events per 10,000 participants with CI between 0 to 27.

What it also ironic about the attack on the Doshi paper is that, just before her concluding remarks and ball juggling, Susan uncritically cites a very flawed modelling study crediting nearly 20 millions of lives saved to Covid jabs as a "rebuttal."

Bayesian analysis of the data

The benefits of applying a Bayesian analysis to the data is that we are able to ‘learn’ the full probability distributions of the adverse reaction rates for vaccine and placebo. This enables us not just to compute the risk ratios and CIs (we get slightly different results to Doshi) but, crucially, also to make explicit probabilistic statements about whether the vaccine SAE rate is higher than that of the placebo (this approach is the Bayesian alternative to the flawed p-value approach). The results (which we provide below) do indeed provide explicit support for the hypothesis that the SAE rate for vaccine is higher than that of the placebo.

 Serious adverse events

 

Vacc AEs

Placebo AEs

P(Vacc AE) Median & CI

P(placebo AE) Median & CI

RR Median & CI

Prob (vacc higher SAE)

Pfizer

127/18801

93/18785

0.0068 (0.0057, 0.0080)

0.0050 (0.0040, 0.0060)

1.362 (1.044, 1.784)

98.86%

Moderna

206/15185

196/15166

0.01361 (0.0118, 0.0155)

0.01297 (0.0112, 0.0148)

1.050

(0.864, 1.275)

68.76%

Combined

333/33986

289/33951

0.0098 (0.0088, 0.0109)

0.0085 (0.0076, 0.0096)

1.151 (0.983, 1.348)

96.03%

 

Serious adverse events of special interest

 

Vacc AEs

Placebo AEs

P(Vacc AE) Median & CI

P(placebo AE) Median & CI

RR Median & CI

Prob (vacc higher SAE)

Pfizer

52/18801

33/18785

0.00280 (0.0021, 0.0036)

0.00180 (0.0013, 0.0025)

1.56 (1.016, 2.44)

97.92%

Moderna

87/15185

64/15166

0.0058 (0.047, 0.0071)

0.0043 (0.0033, 0.0054)

1.37

(0.98, 1.88)

96.85%

Combined

139/33986

97/33951

0.0041 (0.0035, 0.0048)

0.0029 (0.0023, 0.0035)

1.43 (1.104, 1.857)

99.65%

 

Serious adverse events of special interest matching Brighton’s list

 

Vacc AEs

Placebo AEs

P(Vacc AE) Median & CI

P(placebo AE) Median & CI

RR Median & CI

Prob (vacc higher SAE)

Pfizer

39/18801

28/18785

0.0021 (0.0015, 0.0028)

0.0015 (0.0010, 0.0022)

1.38 (0.86, 2.26)

90.82%

Moderna

65/15185

56/15166

0.0043 (0.0035, 0.0048)

0.0037 (0.0029, 0.0048)

1.16

(0.81, 1.66)

79.04%

Combined

104/33986

84/33951

0.00308 (0.0025, 0.00371)

0.00249 (0.002, 0.0031)

1.24  (0.93, 1.65)

92.57%

 

 

Postscrip: the vicious campaign against Peter Doshi

Following a tweet by Norman Fenton criticising the video blue checkmark ‘surgeon/scientist’  David Gorski made several replies supporting the claims of the video and then made this attack of Peter Doshi.

 



to which blue checkmark Steve Salzberg (“Bloomberg Distinguished Professor of BME, CS, and Biostats at Johns Hopkins University”) replied:

 


 and was supported by Art Caplan - Professor of bioethics:

 

 

References

[1] Fraiman, J., Erviti, J., Jones, M., Greenland, S., Whelan, P., Kaplan, R. M., & Doshi, P. (2022). Serious Adverse Events of Special Interest Following mRNA Vaccination in Randomized Trials. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4125239

[2] Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD (2015) The Extent and Consequences of P-Hacking in Science. PLoS Biol 13(3): e1002106. https://doi.org/10.1371/journal.pbio.1002106

[3] “A simple example of Bayesian hypothesis testing”, https://youtu.be/s4yCu__18Jo


15 comments:

  1. Wonderful defence of science. Art Caplan has been a close minded dinosaur forever ready to defend the status quo. When you have a multi billion dollar companies (some of whom have a history of misleading the public) making a tonne of money on their products they deserve a bit of scrutiny. Why anyone wants to jump to the defence of the big Pharma industry types over the small guys is mind blowing. Esp when many of these jabs are mandated by science illiterate politicians. Keep up the great work, appreciate you quest for honesty, diligence, and integrity.

    ReplyDelete
    Replies
    1. Hear, hear! Fenton, Neil et al. are true scientists whose data interpretations using Bayesian modelling confound traditional statistical analysis.

      Delete
  2. This clown is not a scientist and isn't defending science at all, he knows nothing of science.

    ReplyDelete
  3. "If the authors had been “p-hacking” they would have chosen a p-value like 0.05 and would have added, for each comparison of vaccine v placebo"

    This is an example of incredible ignorance of statistics and the breadth of what is identified as p-hacking.

    Do not listen to this blogger, he's defending Doshi because he's just an anti-vaxxer.

    ReplyDelete
  4. Anti-Vaxxer when you have to use these pejorative phrases to try to win an argument, 'You have already lost'. If you are a paid pharma hack, a 77th brigade traitor, a uninformed or misinformed 'useful idiot', either way, people who have the ability to critically think can see it a mile away. TRY HARDER!!

    ReplyDelete
    Replies
    1. Godwin's Law should be expanded to include the pejorative "anti-vaxxer."

      Delete
  5. Dear reader, do not listen to Anonymous person. He disagrees with this blogger because the blogger 'is defending Doshi', and thus regards either Doshi or the blogger (it's not clear whom, perhaps both) to be an anti-vaxxer, which could be interpreted as just another way of saying that Anonymous person strongly disagrees with him (or Doshi, or the other him, it's not clear, or perhaps both of them) about highly contentious issues in biomedical science, but is really extending to you a rather forceful invitation, by no more than the application of a puerile pejorative label, to regard him (not the Anonymous person), or Doshi, or the other him, or both of them (without a single moment's further consideration), as champagne nincompoops and idiots of such extraordinary stupidity that merely giving the slightest credence to their opinions, even if solely for the purpose of rational examination in the course of civil discourse, will put you, dear reader, in jeopardy of infection with an incurable and ghastly form of intellectual leprosy. For those reasons, dear reader, it is best that you do not listen to the Anonymous person, and for Anonymous person to remain as such.

    ReplyDelete
  6. I wish to clarify, dear reader, that in my earlier comment I was referring to the other Anonymous person, and not the most recent Anonymous person, but I feel sure that you, dear reader, will have the good sense to understand which Anonymous person was the topic of my remarks.

    ReplyDelete
  7. Funny how the attacks from these alleged scientists speak nothing about the actual reality of the calculations. They just use misdirection of focus and provide nothing to the contrary. These hacks are the reason the jab was created, to weed out the unworthy and mentally challenged. Hope they just continue to boost!!

    ReplyDelete
  8. Unfortunately, I have forgotten what statistics I once knew, and I never learned enough to be qualified to opine on the merits of this debate.
    That said, the tweet by Salzburg is absolutely unbelievable!
    Basically: "Hey, I'm personal friends with the executive editor so I can probably call in a personal favor and have this guy (Doshi) whose views we disagree with removed from his associate editorship. Who's on board?"
    And he gets immediate response.
    These people (Salzburg and Caplan) are not scientists at all, regardless of their credentials. They are perfect examples of why "lay-people" do not -- and definitely SHOULD NOT -- just "trust the experts" or "trust the science".
    Again, they're not scientists, but rather ideologically/politically motivated highschool bullies. They are, instead, psychologically and emotionally retarded such that they live in a permanent adolescence. Revenge of the Nerds isolating and picking on their designated "unpopular kid", Doshi.
    Astounding.

    ReplyDelete
  9. It’s amazing how many smart people were stupid enough to be willing lab rats for a completely new medical experiment. An experiment that not only does not prevent infection or transmission of disease, it spawns mutations and new variants. And, oh yeah, it has killed more people than Covid, and permanently harmed millions more.

    ReplyDelete
  10. Hi everyone reading this amazing testimony of mine and i want to give my own verdict on how you can win the lottery by contacting Dr Kumar for your lottery numbers. I started playing lottery games for years now and I have never won big. I went online to seek help to see if i could get some tips on how I can win bigger amounts on the lottery and I saw some nice reviews about Dr Kumar who has made different people huge winners in the lottery by sending them lottery numbers. I gave it a try and I contacted Dr Kumar by email and he told me how and what to do before I can become a big lotto winner and I accepted. He made special prayers for me at his temple and sent me the required lottery numbers to play and when I played it, I won a massive $235.4 million POWERBALL jackpot. I was so happy and I chose to receive my winnings in a one-time, lump-sum payment of $160,038,447.27 at Florida Lottery Headquarters in Tallahassee and the balance was given to the store that I bought & played the game from. I would love other people who want to win the lottery to seek help from Dr Kumar and he will also send you the lottery numbers and make you a winner just like me. You can reach Dr Kumar on email: spellcasttemple@gmail.com or WhatsApp +2347051705853 

    ReplyDelete
    Replies
    1. Amparo Parnell, congratulations on your winnings. Please, keep this secret to yourself, otherwise everybody who believe you will be winning the lottery all the time. Also, congratulations on silencing the adults in the room in this comment section.

      Delete
  11. I have been checking out a few of your stories and i can claim pretty clever stuff. I will surely bookmark your website.

    https://truxgo.net/blogs/299122/967409/a-mixed-file-might-be-the-reason-why-your-credit-score-goes-dow

    ReplyDelete
  12. Dr Susan Oliver has responded to your blog post and the claims within it on a video (see below). I'm not qualified to judge whose analysis is correct, but she puts forward a reasonable, reasoned and evidenced case. In particular, she claims you misinterpret the nature of P-hacking, and ignore her central claims about the 'apples vs oranges' comparison problem. What's your response Dr Fenton? https://youtu.be/RvxqyCZlKO0

    ReplyDelete