Now that the vaccines have been around for some time there are a few more studies being published about their effectiveness. Do these studies tell us what we really want to know? Does having a vaccination reduce your chances of serious illness, hospitalisation or death?
The first paper to look at is Comparison of two highly-effective mRNA vaccines for COVID-19 during periods of Alpha and Delta variant prevalence.
This one looked really good as they exactly matched 3 cohorts of around 25,000 people by race, sex, age, location etc etc to begin their comparison and compared these outcomes for subjects with vax 1, vax 2 and unvax:
- SARS-CoV-2 infection: at least one positive SARS-CoV-2 PCR test. The date of infection was taken as the date of the first positive test.
- COVID-19 associated hospitalization: admission to the hospital occurring within 21 days after SARS-CoV-2 infection.
- COVID-19 associated ICU admission: admission to the intensive care unit (ICU) occurring within 21 days after SARS-CoV-2 infection.
- COVID-19 associated mortality: death occurring within 28 days after SARS-CoV-2 infection.
- Breakthrough infection: a SARS-CoV-2 infection occurring after full vaccination (i.e., at least 14 days after the second dose of mRNA-1273 or BNT162b2).
So far, so good. Plenty of subjects in the survey and a range of ages etc. At first glance, it seems to say that the unvaccinated are worse off.
But the study design flaw seems to be that they are not treating the results as a “funnel”. If you go to the ICU, then necessarily you would have been hospitalised. And in order to be hospitalised, you have to have been infected. But the study reports hospitalisations for vaccinated people as a fraction of the original population, not as a fraction of cases.
Take a simple example. Let’s say the vaccine reduces your chances of getting COVID 20 times, and we have the following outcomes:
- Unvax: Pop: 1,000,000; COVID: 20,000; Hospital: 1,000; ICU: 500; Dead: 100
- Vax 1: Pop: 1,000,000; COVID: 1,000; Hospital: 50; ICU: 25; Dead: 5Ta
- Vax 2: Pop: 1,000,000; COVID: 1,000; Hospital: 100; ICU: 75; Dead:
For Vax 1, the only ratio that changed was Pop/COVID; everything else is the same. But it gets reported as “Vax reduces your chance of dying by 20x”. Yes… but… if you actually get COVID after vaccination, your odds (of hospital or death) are still exactly the same!
Do most people understand it this way?
But Vax 2 is worse! If you contract COVID after having Vax 2, your chances of hospitalisation or death are actually worse (at that point) than the unvaccinated population! Is that because only the most at-risk people still get COVID after vaccination? Or does it makes side-effects worse?
Finally, in practice, we seldom have big enough studies/numbers to draw useful conclusions. In this study, even though they began with a significant number of subjects, it’s 1/6 vs 2/11, vs 17/82 who went into ICU after hospitalisation for COVID. Those all seem roughly the same ratios. So any effect seems purely driven by reduction at the top of the funnel (which is still great!) but if vaccines were 95% effective it would be great to see that at each stage in the funnel, not just a flow-on result from the “top of funnel” reduction.
Going back to this study, the rates are shown as an order-of-magnitude lower for vax (first two) than unvax… but the ratio vs the previous line isn’t significantly different in either case: PCR->Hospital, Hospital->ICU.
Overall, it reduces the chances of serious illness (so it’s not a false claim), but that’s possibly purely an effect of reducing COVID infection in the first place. If you do get COVID, it’s a 15% vs 25% chance that you’ll go to hospital, vax or not. (6 of 38 vs 82 of 321). But the numbers are so small here it is unwise to extrapolate much. (1 person out of 38 is a 2.6% change.)
Tables 3 and 4 of the study break down stats by month. In July, Moderna (the more effective vaccine) was 3 hospitalisations out of 15 breakthrough infections (=20%) and unvax was 18 out of 73 (25%). Again, when people see simplistic headlines “Vaccines protect against serious illness”, they’re likely to think “I won’t go to hospital if I get COVID” but it seems that your chances (once you’re infected) are about the same whether you had the vaccine or not.
The next paper is one from Israel so all about Pfizer. BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting
All persons who were newly vaccinated during the period from December 20, 2020, to February 1, 2021, were matched to unvaccinated controls in a 1:1 ratio according to demographic and clinical characteristics.
Started with 596,618 people, so that sounds like a promising sample size.
However… when one digs into the numbers, not everything seems to be what it appears. One issue seems to be that people kept getting vaccinated!
Data for 44% of the unvaccinated controls and their matched pairs were censored when the controls received the vaccine.
It would be unethical (I assume) to prevent people from being vaccinated during the study, so as soon as they did, that pair disappeared from the study? So there’s almost half that number gone already.
Based on Figure 2, they started with 596,618 matched people at day 0, but only had ~4000 matched people at day 42. That’s a much smaller study.
As for the first study, everything is measured as a comparison of rate-per-1000-person-days of the whole group:
In the follow-up period starting 7 days after the second dose, the vaccine effectiveness for documented infections, symptomatic illness, hospitalization, and severe disease was 92% (95% CI, 88 to 95), 94% (95% CI, 87 to 98), 87% (95% CI, 55 to 100), and 92% (95% CI, 75 to 100), respectively.
As noted above, these numbers appear to be completely driven by a reduction in documented infection (since each more serious state can’t happen without all the previous ones).
Extracting numbers from the tables gives this:
| Unvaccinated | Day 28 | Day 42 | Difference | Fraction | Percentage of infected |
| Documented Infection | 5775 | 6100 | 325 | 100.0% | |
| Symptomatic | 3433 | 3607 | 174 | 53.5% | 53.5% |
| Hospitalisation | 244 | 259 | 15 | 8.6% | 4.6% |
| Severe | 157 | 174 | 17 | 113.3% | 5.2% |
| Death | 27 | 32 | 5 | 29.4% | 1.5% |
| Vaccinated | Day 28 | Day 42 | Difference | Fraction | Percentage of infected |
| Documented Infection | 4405 | 4460 | 55 | 100.00% | |
| Symptomatic | 2373 | 2389 | 16 | 29.1% | 29.09% |
| Hospitalisation | 108 | 110 | 2 | 12.5% | 3.64% |
| Severe | 52 | 55 | 3 | 150.0% | 5.45% |
| Death | 7 | 9 | 2 | 66.7% | 3.64% |
Having done this we can’t compare them directly because we don’t know that these people are matched at all! They could be different ages, sex, comorbidity, etc.
But, just by raw numbers, 3.5% of vax people who had a documented infection ended up dying, but only 1.5% of unvax people. As before, the concern is that media presents vaccines as reducing your chance of death (which, I will repeat, is true!) but does present an incorrect picture for people who do contract COVID post-vaccination.
“Vaccines protect against severe illness and death” is misleading because I think people fill in “if you get COVID” to the end of that phrase.
So far the studies do not appear to bear that out. Once you get COVID there is not a huge difference in outcomes of hospitalisation or death. The vaccine may prevent you from getting COVID in the first place but that seems to be where the benefit stops.