SARS-CoV-2/COVID-19: How can we tell whether a vaccine works?

  • Efficacy in the elderly population for this vaccine was 94%
  • 10 severe cases of COVID-19 occurred, with 9 in the placebo group and 1 in the vaccinated
  • Short-term adverse events from the vaccine candidate were based on 8000 patients and deemed to be generally mild and self-limiting (see below for clarification on terminology) with fatigue and headache being the most common grade 3 adverse events at ~4% and 2% of vaccinees respectively.
  • Of the 95 cases, 5 were in the vaccinated group and 90 were in the unvaccinated group
  • 15 cases occurred in older adults
  • 20 of the cases occurred in individuals from diverse backgrounds: 12 Hispanic or LatinX, 4 Black/African-American, 3 Asian-Americans, 1 multiracial
  • 11 of the cases were severe. The exciting part: all of them occurred in the placebo group.
  • The vaccine candidate itself was generally well-tolerated with severe adverse events being rare and what one would expect from a vaccine i.e. injection site pain, headache, fever, redness, swelling. These were generally short-lived, which bodes well.
  • Severe adverse event and serious adverse event (SAE) are NOT the same thing. Serious adverse events are those which are life-threatening, life-limiting, or medically important in some way (e.g. requiring a prolonged hospital stay). Severe adverse events are not a true regulatory term per se, but generally it refers to a Grade 3 event as defined by the CBER industry guidance.
  • Generally, severe per the trial protocols of both Moderna and Pfizer, refers to the adverse event causing some level of interference with daily life or requires medical intervention. It is not a comment on the duration of that adverse effect which both companies report were generally short-lived. If for instance, you developed a fever that caused significant enough discomfort that you would take acetaminophen to lower it, the fever is considered a severe adverse event. Note that this also does not equate to an truly dangerous outcome (e.g. requiring urgent medical care, hospitalization, or causing prolonged disability), as this would be a grade 4 event.
  • Adverse event and adverse reaction are not synonyms. An adverse event is anything that happens in the course of a clinical trial regardless of whether or not it is caused by the product (adverse events are recorded for the placebo group as well). Adverse reactions are those adverse events which are likely to have a causal relationship to the receipt of the pharmaceutical. For example, if someone in a trial receives a vaccine (or placebo) and subsequently experiences a gun shot wound, this would be documented as an adverse event on the package insert. I cannot conceive of a scenario in which receipt of a vaccine could actually cause a gunshot wound so I will make the epistemic leap that these two things are not causally related. For those who think I am being facetious, this is a real adverse event documented on the package insert for Gardasil (though the individual in question was in the placebo group).
Deaths in the Gardasil vaccine trials. See page 8. Many methods exist to determine whether or not there is a causal relationship between an adverse event and the pharmaceutical in question. Broadly, there are those based on expert opinion, algorithms, and probabilistic methods. There is a good discussion here, but it is paywalled.

But what does it mean for a vaccine to be effective?

Vaccine effectiveness is defined with respect to a given phenomenon. For instance, vaccines can (rarely) protect against infection entirely (see text for details), but measuring this properly is cumbersome and this is not necessary for a successful campaign. With bacteria, vaccines can have some protection against colonization of the pathogen at the relevant anatomical site. Some vaccines are unable to completely shield from disease but can prevent the worst outcomes. In addition to this, vaccines confer protection beyond the vaccinee (see text for details). Halloran M, Longini I, Struchiner C. Design and Analysis of Vaccine Studies. Springer Science; 2010. Table 2.1
DESIGN OF VACCINE EFFICACY TRIALS TO BE USED DURING PUBLIC HEALTH EMERGENCIES — POINTS OF CONSIDERATIONS AND KEY PRINCIPLES. Who.int. Figure 2
N. E. Dean, P.-S. Gsell, R. Brookmeyer, V. De Gruttola, C. A. Donnelly, M. E. Halloran, M. Jasseh, M. Nason, X. Riveros, C. H. Watson, A. M. Henao-Restrepo, I. M. Longini, Design of vaccine efficacy trials during public health emergencies. Sci. Transl. Med. 11, eaat0360 (2019). Figure 1
O’neill R. On sample sizes to estimate the protective efficacy of a vaccine. Statistics in Medicine. 1988;7(12):1279–1288.

Will these vaccines protect against severe COVID-19?

There have also been several opinion pieces that have criticized clinical trials of SARS-CoV-2 vaccine candidates for not examining whether or not the vaccines will prevent severe disease, and therefore the vaccines will not be able to answer the most important question. COVID-19 in most cases is a relatively self-limiting infection, sometimes even asymptomatic, and so the argument goes that a vaccine that is able to achieve prevention of mild disease may not be able to prevent severe disease, which would not be the public health victory that some are heralding the success of Pfizer and Moderna’s efficacy trials to be. If this were solely about biostatistical considerations, that argument would certainly be formidable. However, it is weakened greatly when one factors in the immunology and dynamics of infectious disease inherent to the problem. The case is well-illustrated with influenza vaccines. Yes, these vaccines, as discussed above, are often unable to prevent disease completely. But, the vaccines work quite well for preventing the worst outcomes of influenza infection (some of the publications supporting this can be found here). This suggests that it is in fact harder to prevent mild cases than severe ones, and this in fact makes sense from the perspective of the immunology involved.

What don’t we know yet?

Durability of the vaccine candidates remains a critical open question. There’s no way to know this without waiting long enough for vaccinated people to start getting sick again, unfortunately, but there is good reason to be optimistic on this count, as a recent study suggests that immunity to SARS-CoV-2 following infection is likely long-lived, even in spite of the complex antagonism of the immune system by SARS-CoV-2.

References

  1. N. E. Dean, P.-S. Gsell, R. Brookmeyer, V. De Gruttola, C. A. Donnelly, M. E. Halloran, M. Jasseh, M. Nason, X. Riveros, C. H. Watson, A. M. Henao-Restrepo, I. M. Longini, Design of vaccine efficacy trials during public health emergencies. Sci. Transl. Med. 11, eaat0360 (2019).
  2. DESIGN OF VACCINE EFFICACY TRIALS TO BE USED DURING PUBLIC HEALTH EMERGENCIES — POINTS OF CONSIDERATIONS AND KEY PRINCIPLES. Who.int.
  3. Halloran M, Longini I, Struchiner C. Design and Analysis of Vaccine Studies. Springer Science+Business Media; 2010.
  4. O’neill R. On sample sizes to estimate the protective efficacy of a vaccine. Statistics in Medicine. 1988;7(12):1279–1288.
  5. Pfizer and BioNTech Conclude Phase 3 Study of COVID-19 Vaccine Candidate, Meeting All Primary Efficacy Endpoints | Pfizer. Pfizer.com. 2020.
  6. Analyzing Phase 3 COVID-19 Vaccine Trials | School of Public Health | University of Illinois at Chicago. Publichealth.uic.edu. 2020.
  7. Dan J, Mateus J, Kato Y, Hastie K, Faliti C, Ramirez S, Frazier A, Yu E, Grifoni A, Rawlings S et al. Immunological memory to SARS-CoV-2 assessed for greater than six months after infection. biorxiv. 2020.
  8. Chu H, Halloran M. Bayesian estimation of vaccine efficacy. Clinical Trials. 2004;1:306–314.
  9. Gilbert P, Janes H, Huang Y. Power/Sample Size Calculations for Assessing Correlates of Risk in Clinical Efficacy Trials. Stat Med. 2016;35(21):3745–3759.
  10. Moderna’s COVID-19 Vaccine Candidate Meets its Primary Efficacy Endpoint in the First Interim Analysis of the Phase 3 COVE Study | Moderna, Inc. Moderna, Inc. 2020.
  11. A Phase 3, Randomized, Stratified, Observer-Blind, Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older. Modernatx.com. 2020.
  12. A PHASE 1/2/3, PLACEBO-CONTROLLED, RANDOMIZED, OBSERVER-BLIND, DOSE-FINDING STUDY TO EVALUATE THE SAFETY, TOLERABILITY, IMMUNOGENICITY, AND EFFICACY OF SARS-COV-2 RNA VACCINE CANDIDATES AGAINST COVID-19 IN HEALTHY INDIVIDUALS. Pfe-pfizercom-d8-prod.s3.amazonaws.com. 2020.
  13. Kucharski A, Eggo R, Watson C, Camacho A, Funk S, Edmunds W. Effectiveness of Ring Vaccination as Control Strategy for Ebola Virus Disease. Emerging Infectious Diseases. 2016;22(1):105–108.
  14. B. D. Kirkpatrick, S. S. Whitehead, K. K. Pierce, C. M. Tibery, P. L. Grier, N. A. Hynes, C. J. Larsson, B. P. Sabundayo, K. R. Talaat, A. Janiak, M. P. Carmolli, C. J. Luke, S. A. Diehl, A. P. Durbin, The live attenuated dengue vaccine TV003 elicits complete protection against dengue in a human challenge model. Sci. Transl. Med. 8, 330ra36 (2016).
  15. Human Challenge Trials for Vaccine Development: regulatory considerations. Who.int. 2016.
  16. Will a small, long-shot U.S. company end up producing the best coronavirus vaccine?. 2020 [accessed 2020 Nov 19]. https://www.sciencemag.org/news/2020/11/will-small-long-shot-us-company-end-producing-best-coronavirus-vaccine
  17. Pinkbook | Varicella | Epidemiology of Vaccine Preventable Diseases | CDC. Cdc.gov. 2020.
  18. Guidelines on clinical evaluation of vaccines: regulatory expectations. Who.int. 2016.
  19. Nauta J. Statistics in Clinical Vaccine Trials. Springer-Verlag Berlin Heidelberg; 2011.
  20. Clemens J. Evaluating new vaccines for developing countries. Efficacy or effectiveness?. JAMA: The Journal of the American Medical Association. 1996;275(5):390–397.
  21. Statistical Considerations for Clinical Trials During the COVID-19 Public Health Emergency Guidance for Industry. Fda.gov. 2020.
  22. Development and Licensure of Vaccines to Prevent COVID-19 Guidance for Industry. Fda.gov. 2020.
  23. Halloran M, Longini I, Struchiner C. Design and Interpretation of Vaccine Field Studies. Epidemiologic Reviews. 1999;21(1):73–88.
  24. Agbabiaka, T.B., Savović, J. & Ernst, E. Methods for Causality Assessment of Adverse Drug Reactions. Drug-Safety 31, 21–37 (2008). https://doi.org/10.2165/00002018-200831010-00003
  25. Iannelli MD V. How Often Should You Do Vaccine Titer Testing? — VAXOPEDIA. VAXOPEDIA. 2019.
  26. Tokars J, Patel M, Foppa I, Reed C, Fry A, Ferdinands J. Waning of Measured Influenza Vaccine Effectiveness Over Time: The Potential Contribution of Leaky Vaccine Effect. Clinical Infectious Diseases. 2020.
  27. Human Challenge Trials for Vaccine Development: regulatory considerations. Who.int. 2016.
  28. Keech C, Albert G, Cho I, Robertson A, Reed P, Neal S, Plested J, Zhu M, Cloney-Clark S, Zhou H et al. Phase 1–2 Trial of a SARS-CoV-2 Recombinant Spike Protein Nanoparticle Vaccine. New England Journal of Medicine. 2020.
  29. Toxicity Grading Scale for Healthy Adult and Adolescent Volunteers Enrolled in Preventive Vaccine Clinical Trials. Fda.gov. 2020.
  30. Benefits of Influenza Vaccination: Selected Publications | CDC. Cdc.gov. 2016.
  31. Addetia A, Crawford K, Dingens A, Zhu H, Roychoudhury P, Huang M, Jerome K, Bloom J, Greninger A. Neutralizing Antibodies Correlate with Protection from SARS-CoV-2 in Humans during a Fishery Vessel Outbreak with a High Attack Rate. Journal of Clinical Microbiology. 2020;58(11).
  32. Gupta, A., Madhavan, M.V., Sehgal, K. et al. Extrapulmonary manifestations of COVID-19. Nat Med 26, 1017–1032 (2020). https://doi.org/10.1038/s41591-020-0968-3
  33. Fajnzylber, J., Regan, J., Coxen, K. et al. SARS-CoV-2 viral load is associated with increased disease severity and mortality. Nat Commun 11, 5493 (2020). https://doi.org/10.1038/s41467-020-19057-5
  34. Lindner D, Fitzek A, Bräuninger H, Aleshcheva G, Edler C, Meissner K, Scherschel K, Kirchhof P, Escher F, Schultheiss H et al. Association of Cardiac Infection With SARS-CoV-2 in Confirmed COVID-19 Autopsy Cases. JAMA Cardiology. 2020;5(11):1281.
  35. Puelles V, Lütgehetmann M, Lindenmeyer M, Sperhake J, Wong M, Allweiss L, Chilla S, Heinemann A, Wanner N, Liu S et al. Multiorgan and Renal Tropism of SARS-CoV-2. New England Journal of Medicine. 2020;383(6):590–592.
  36. Neuroinvasion of SARS-CoV-2 in human and mouse brain. REGENHEALTHSOLUTIONS (RHS). 2020.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Edward Nirenberg

Edward Nirenberg

I write about vaccines here. You can find me on Twitter @enirenberg and at deplatformdisease.com (where I publish the same content without a paywall)