SARS-CoV-2/COVID-19: How can we tell whether a vaccine works?

The Short Version: There are multiple vaccines coming out for COVID-19 demonstrating excellent efficacy in Phase III trials. They have also been shown to be well-tolerated with only transient unpleasant side effects. Vaccine efficacy and vaccine effectiveness are different- efficacy refers to the performance of the vaccine in the clinical trial while effectiveness is how well it does in the real world. Even though these vaccines have excellent efficacy, we still don’t yet know how effective they are. We also don’t understand how well they can interrupt transmission, so unfortunately even after the vaccines come out, we will still have to continue public health measures for some time.

Recently, Pfizer announced that its SARS-CoV-2/COVID-19 vaccine candidate met its efficacy endpoint at 95% based on 170 cases, in which 8 occurred in the vaccinated group and 162 in the unvaccinated group, which is excellent news. We also know:

• Efficacy in the elderly population for this vaccine was 94%
• 10 severe cases of COVID-19 occurred, with 9 in the placebo group and 1 in the vaccinated
• Short-term adverse events from the vaccine candidate were based on 8000 patients and deemed to be generally mild and self-limiting (see below for clarification on terminology) with fatigue and headache being the most common grade 3 adverse events at ~4% and 2% of vaccinees respectively.

Unfortunately, they weren’t very charitable with their details.

Regarding Moderna, their interim analysis showed that its candidate was 94.5% effective based on its interim analysis and also gave some more details:

• Of the 95 cases, 5 were in the vaccinated group and 90 were in the unvaccinated group
• 15 cases occurred in older adults
• 20 of the cases occurred in individuals from diverse backgrounds: 12 Hispanic or LatinX, 4 Black/African-American, 3 Asian-Americans, 1 multiracial
• 11 of the cases were severe. The exciting part: all of them occurred in the placebo group.
• The vaccine candidate itself was generally well-tolerated with severe adverse events being rare and what one would expect from a vaccine i.e. injection site pain, headache, fever, redness, swelling. These were generally short-lived, which bodes well.

Please note, however:

• Severe adverse event and serious adverse event (SAE) are NOT the same thing. Serious adverse events are those which are life-threatening, life-limiting, or medically important in some way (e.g. requiring a prolonged hospital stay). Severe adverse events are not a true regulatory term per se, but generally it refers to a Grade 3 event as defined by the CBER industry guidance.
• Generally, severe per the trial protocols of both Moderna and Pfizer, refers to the adverse event causing some level of interference with daily life or requires medical intervention. It is not a comment on the duration of that adverse effect which both companies report were generally short-lived. If for instance, you developed a fever that caused significant enough discomfort that you would take acetaminophen to lower it, the fever is considered a severe adverse event. Note that this also does not equate to an truly dangerous outcome (e.g. requiring urgent medical care, hospitalization, or causing prolonged disability), as this would be a grade 4 event.
• Adverse event and adverse reaction are not synonyms. An adverse event is anything that happens in the course of a clinical trial regardless of whether or not it is caused by the product (adverse events are recorded for the placebo group as well). Adverse reactions are those adverse events which are likely to have a causal relationship to the receipt of the pharmaceutical. For example, if someone in a trial receives a vaccine (or placebo) and subsequently experiences a gun shot wound, this would be documented as an adverse event on the package insert. I cannot conceive of a scenario in which receipt of a vaccine could actually cause a gunshot wound so I will make the epistemic leap that these two things are not causally related. For those who think I am being facetious, this is a real adverse event documented on the package insert for Gardasil (though the individual in question was in the placebo group).

Deaths in the Gardasil vaccine trials. See page 8. Many methods exist to determine whether or not there is a causal relationship between an adverse event and the pharmaceutical in question. Broadly, there are those based on expert opinion, algorithms, and probabilistic methods. There is a good discussion here, but it is paywalled.

These announcements led to some renewed optimism and hope, even as in the US, COVID-19 attained catastrophic levels (which actually resulted in Pfizer and Moderna failing to conduct earlier planned interim analyses, because cases rose so precipitously), and indeed, though questions linger it is cause for optimism.

But what does it mean for a vaccine to be effective?

Vaccine effectiveness is defined with respect to a given phenomenon. For instance, vaccines can (rarely) protect against infection entirely (see text for details), but measuring this properly is cumbersome and this is not necessary for a successful campaign. With bacteria, vaccines can have some protection against colonization of the pathogen at the relevant anatomical site. Some vaccines are unable to completely shield from disease but can prevent the worst outcomes. In addition to this, vaccines confer protection beyond the vaccinee (see text for details). Halloran M, Longini I, Struchiner C. Design and Analysis of Vaccine Studies. Springer Science; 2010. Table 2.1

This is already deceptively complicated because vaccine effectiveness and vaccine efficacy are NOT synonymous and the difference is not semantical (so make sure you use the right one). Phase III studies are typically efficacy trials: they are formal explorations of whether or not a vaccine works with rigorous monitoring involving laboratory evaluation of immunogenicity and of course, infection. When examined in the real world, we typically cannot attain such a rigorous standard of exploration regarding a vaccine. Cases will happen (or they won’t) but titers are rarely necessary for most practical circumstances and most assuredly, no one is evaluating immunogenicity. Effectiveness trials occur under real-world conditions so to speak. In general, phase IV studies are effectiveness trials. Because efficacy trials tend to be much more labor-intensive and expensive, there are circumstances in which effectiveness trials are considered appropriate substitutes, especially after efficacy has already been established. The evaluation of influenza vaccines each season with respect to their ability to prevent ILI (influenza-like illness) is an example of a vaccine effectiveness study. As you can see from the table on your left though, there are a lot of complexities to the concepts of vaccine effectiveness and efficacy (Design and Analysis of Vaccine Studies is an excellent monograph and I encourage people to read it for more detailed information). In general, effectiveness of any vaccine is less than its efficacy, as the immunity from any vaccine will generally wane after some amount of time, hence requiring a “booster.” Furthermore, though we now have a pretty good idea of whether or not Pfizer and Moderna’s vaccines are efficacious, we still know nothing, in the formal sense, about their effectiveness.

DESIGN OF VACCINE EFFICACY TRIALS TO BE USED DURING PUBLIC HEALTH EMERGENCIES — POINTS OF CONSIDERATIONS AND KEY PRINCIPLES. Who.int. Figure 2

Usually when people think about vaccine efficacy, they want to know how well a vaccine can prevent infection. The truth is, this isn’t how most vaccines work at all. It is generally very rare for a vaccine to be able to altogether prevent infection, though there are some that do e.g. HPV. The confusion here comes from a conflation of the notions of “infection” and “disease.” As is well demonstrated by the existence of asymptomatic and presymptomatic cases, these are not the same thing. One could readily have an infection and not have any overt disease. More precisely, the moment a virus enters a cell, that cell is infected, and the host has an infection. If a host is able to prevent even infection from occurring, they are said to have sterilizing immunity, a privileged immunological status mediated primarily by antibodies that can bind and neutralize the pathogen before infection can happen. To be clear, if a single viral particle so much as enters a single cell, that’s infection. The corresponding vaccine efficacy measure is the measure of susceptibility (to infection). However, we generally don’t care about how well a vaccine prevents infection because vaccination does not have to prevent infection to save lives, and to hold them to such a standard in general is unreasonable, as even most infections do not confer sterilizing immunity. Still, the language in the literature is at times imprecise with their language. In general, however, unless examining vaccinees free of disease for presence of the pathogen of interest e.g. PCR on asymptomatic individuals in the trial to confirm no presence of virus, the ability to prevent infection in the vaccinee is not being examined. There are many other vaccine effects that we do care about. For instance, a primary concern is whether or not the vaccine protects against the development of disease. Sometimes, vaccines are leaky- unable to prevent disease altogether, but reliably preventing the worst outcomes of disease e.g. pneumonia, sepsis, the need for ICU care. One example of is influenza, and the metric to evaluate how well it does this is VEp, vaccine efficacy for progression or pathogenicity (see the table at the left).

N. E. Dean, P.-S. Gsell, R. Brookmeyer, V. De Gruttola, C. A. Donnelly, M. E. Halloran, M. Jasseh, M. Nason, X. Riveros, C. H. Watson, A. M. Henao-Restrepo, I. M. Longini, Design of vaccine efficacy trials during public health emergencies. Sci. Transl. Med. 11, eaat0360 (2019). Figure 1

It is critical that a vaccine trial be designed with the correct endpoint in mind. For instance, if prevention of infection is chosen as an endpoint, almost all vaccines will fail to meet it, because as stated before, that’s not how most vaccines work. Approval of a vaccine is based on whether or not it can successfully meet its primary endpoint. Within any given trial however, there could be many secondary endpoints. For example, most of the clinical trials selected as their primary endpoint the prevention of COVID-19 (the disease, not SARS-CoV-2 infection) of any severity, or reduction in disease severity, a decision that (somewhat unfairly) was criticized by some. Hence the vaccine candidate could be approved only if it was deemed to meet the primary endpoint of reducing or preventing COVID-19.

An important thing to consider in the evaluation of a vaccine’s effectiveness is that there are many different so-called vaccine effects at play. Certainly, you hope that the vaccine protects the vaccinee. But in many cases, having to immunize everyone in a population is unrealistic, but also unnecessary because of indirect effects of vaccination, also called spillover effects. Direct effects of vaccination measure the protection conferred by vaccination to the vaccinee, while indirect effects refer to the protective effects on the people in contact with the vaccinee which result in herd protection effects (herd immunity).

The sum of direct and indirect effects is known as total vaccine effectiveness. This is expected to be greater than the direct effectiveness because, one hopes, indirect effects are positive.

Finally, overall effectiveness compares the incidence of the relevant parameter in the vaccinated group with those who are unvaccinated for any reason, and is analyzed by comparing cluster-level incidences across vaccinated and control populations. This parameter is the most encapsulating parameter in reflecting the impact of a vaccination campaign on public health overall.

Assessing all the different types of effects from vaccines is not straightforward and requires some methodological finagling, mainly having to do with how a trial is randomized (one could assess vaccine efficacy via an observational trial to but this is not ideal). At the most basic level, randomization could occur at the individual level (individual randomized controlled trial, iRCT) — subjects are matched based on as many characteristics as possible to minimize confounders and assigned to be in the placebo or vaccine group at random. This tends to work well because vaccine trials will occur across many sites and the incidence of the disease in question is certain to vary, and this readily allows us to measure the direct vaccine effect (vaccine efficacy). However, if there is a very strong indirect vaccine effect, it can be harder to gather data because of such substantial interruptions to transmission. In some cases, there are multiple vaccine candidates for the same disease to be evaluated in which case a multi-arm iRCT can be done. Randomization can also occur in clusters (cluster randomized controlled trial, cRCT) where subjects in a cluster receive the same intervention (vaccine vs. placebo), which is most useful for assessing the indirect effects of a vaccine. This strategy is closely related to ring vaccination trials, in which those who have had contact with the disease agent are vaccinated (forming a ring), a strategy that has been used to great effect with in particular smallpox, but also Ebola. A hybrid of cRCTs and iRCTs exists in the form of two-stage RCTs where clusters are randomized to a certain level of vaccine coverage and the individuals within the clusters are randomized to receive the vaccine or the placebo, allowing for measurement of both direct and indirect effects of vaccination. Each of these strategies have associated strengths and weaknesses, particularly pertaining to considerations of statistical power, and are summarized in greater depth here.

There are also human challenge studies which are a bit more straightforward, but have profound ethical constraints. There are also some methodological issues. For example, human challenge trials involve individuals at lowest risk for severe disease or death, and these individuals may have important immunological differences compared with those in whom the vaccine may be most important (those at highest risk of bad outcomes from the disease, e.g. the elderly and COVID-19). I have a lot of ambivalence about these as a concept for approval, and I think they are inappropriate for COVID-19, until at a minimum, we are able to create a highly effective therapeutic regimen. Nonetheless, regardless of how I feel, they will be happening.

Once a trial has been conducted, the determination of vaccine efficacy is relatively simple:

O’neill R. On sample sizes to estimate the protective efficacy of a vaccine. Statistics in Medicine. 1988;7(12):1279–1288.

Where VE is the vaccine efficacy (or effectiveness depending on the nature of the trial), ARU is the attack rate of the relevant condition in the unvaccinated group, and ARV is the attack rate of the relevant condition in the vaccinated group. For example, from Pfizer’s data, there were ~42,000 individuals who received 2 doses of the vaccine with ~21,000 individuals in the placebo and ~21,000 in the vaccinated group. The vaccinated group had 8 cases of COVID-19, and the unvaccinated group had 162 cases. Thus:

At preventing COVID-19 of any severity. An analogous calculation shows that the vaccine has an efficacy of 88.9% at preventing severe COVID-19 (9 cases in the unvaccinated group and 1 in the vaccinated group). Importantly, this does not give a true efficacy of the vaccine per se. This is a point estimate, but because this uses real-world data, a confidence interval needs to be constructed to determine the probable range of efficacy values the vaccine has with respect to a given disease. The details for how this is done are discussed here. In short, there needs to be sufficiently many cases that we have the statistical power to reject the null hypothesis (generally, the hypothesis that the intervention has no effect, but it can be altered; for instance the minimum efficacy requirement for any vaccine for COVID-19 was set at 30% by the FDA, so the null hypothesis is that the vaccine is 30% effective or less at reducing the incidence of COVID-19 of any severity) at a given significance level. The confidence interval constructed must be outside what would be expected by the null hypothesis for it to be rejected, meaning in this case, the lower bound of the confidence interval in question must be above 30% for vaccine efficacy.

Studies of vaccine efficacy are event-driven trials, and so despite recruiting huge numbers of individuals (usually at least 30,000 but the polio vaccine had a trial of over 2 million people before approval), the events being analyzed are, relatively speaking, rare. The only way to tell whether or not a vaccine actually works, is, after all, to let people in the trial become sickened with the relevant condition. For example, one of the pre-approval trials of the varicella (chickenpox) vaccine had 39 cases of chickenpox, all of which occurred in the placebo group, and about 900 participants, which gave a vaccine efficacy of 100% after 1 dose. This doubtlessly seems like a very small number to justify the approval of a pharmaceutical (and to be clear- this was not the only study which examined the efficacy of the varicella vaccine prior to it being granted licensure), it’s actually not. Consider however, the secular trends in the incidence of chickenpox if you have doubts about this approach. We can actually take an even more extreme case: TV003 is a novel Dengue vaccine which has not yet gained approval, but was evaluated in a human challenge study (in which 20 individuals received no vaccine and were deliberately infected with Dengue while 21 received the vaccine and were infected with Dengue). There were no cases of Dengue fever in the vaccinated group and all individuals in the unvaccinated group developed the disease.

The reason for the recruitment of huge numbers of people to vaccine trials is several-fold. Firstly, it allows for a more comprehensive safety evaluation in that rarer adverse events will occur with a larger population (though notably it should not be expected that all possible events will occur in a trial, and some especially rare ones, e.g. those at a frequency of 1 per million or less, will be found only in postmarketing surveillance, phase IV), which is a major advantage for large RCTs over human challenge studies. However, the other key reason is because the events are rare, you need huge numbers to see them happen. Consider for instance a vaccine with absolute 100% direct efficacy, significant durability of the protective immune response, and significant indirect efficacy- it could take a very long time to see the number of cases needed for an efficacy determination unless absolutely huge numbers of individuals are enrolled in a trial, as the vaccine will disrupt transmission to a substantial extent. The efficacy evaluation is based on these events, rather than the actual sample size. There’s an important principle of statistics that comes into play here: larger effect sizes allow for smaller samples to demonstrate them. If you examine the effect size of a vaccine compared to virtually any other pharmaceutical in existence, it is almost comical how much better vaccines work. For this reason, far fewer cases are needed for vaccines to demonstrate efficacy.

When I asked Professor Natalie Dean, biostatistician and contributing and sometimes first author to multiple publications I cite in this post, over Twitter about the numbers involved in this trial, she told me that 95 cases (based on the Moderna interim analysis) allow for the construction of a 90% confidence interval that does not contain 30 % (meaning the vaccine would have met its efficacy endpoint in the trial, with a 10% chance of the differences in the vaccine group being due to random chance) assuming a true efficacy of 60%. For a vaccine whose efficacy is 95%, the actual number of cases one needs is much smaller. Thus, at 170 cases with a computed efficacy of 95.1%, we can be very confident that Pfizers vaccine is extremely efficacious, and because their formulations are so similar, we can presume that Moderna’s results will be similarly good. It would be excellent if the results were elaborated upon, preferably in the context of a peer-reviewed publication, but unless there is some gross fabrication at play here, it is extremely likely that Pfizer and Moderna have successfully devised highly efficacious vaccines. What’s more is that though both of these are mRNA vaccines, many other candidates employed a very similar strategy for protecting against COVID-19 by targeting the spike glycoprotein or just its receptor-binding domain (RBD), generating high titers of antibodies, and a Th1-dominated cytokine profile. I personally am extremely interested in the efficacy data for Novavax’s vaccine (which is my personal favorite because the phase 1/2 data were so impressive, but that said, I will gladly take Pfizer or Moderna’s vaccines, should they pass regulatory muster which they almost certainly will).

It would be ideal if these results weren’t by press release, but it’s hard to deny that these represent excellent news. I still have a great deal of ambivalence about a vaccine approved by EUA personally, but it would be understandable why pharmaceutical companies would want to apply for one given these data.

Will these vaccines protect against severe COVID-19?

There have also been several opinion pieces that have criticized clinical trials of SARS-CoV-2 vaccine candidates for not examining whether or not the vaccines will prevent severe disease, and therefore the vaccines will not be able to answer the most important question. COVID-19 in most cases is a relatively self-limiting infection, sometimes even asymptomatic, and so the argument goes that a vaccine that is able to achieve prevention of mild disease may not be able to prevent severe disease, which would not be the public health victory that some are heralding the success of Pfizer and Moderna’s efficacy trials to be. If this were solely about biostatistical considerations, that argument would certainly be formidable. However, it is weakened greatly when one factors in the immunology and dynamics of infectious disease inherent to the problem. The case is well-illustrated with influenza vaccines. Yes, these vaccines, as discussed above, are often unable to prevent disease completely. But, the vaccines work quite well for preventing the worst outcomes of influenza infection (some of the publications supporting this can be found here). This suggests that it is in fact harder to prevent mild cases than severe ones, and this in fact makes sense from the perspective of the immunology involved.

Recall the discussion of sterilizing immunity: it is a response so rapid and so potent that infection does not even occur in the first place. It is exceptionally rare for any immune response, whether precipitated by an infection or vaccine, to attain this privileged immunological status. As a thought experiment, let’s consider successively lowering antibody levels and the outcome on the disease state. At the levels immediately below those required for sterilizing immunity, we expect to see an abortive infection that is well confined to a given anatomical compartment (the one the pathogen enters through), which would likely be asymptomatic as escaping virions are rapidly neutralized. As the intensity of the antibody response declines, the ability of the pathogen to spread increases, resulting in more manifestations of disease to regions outside the primary anatomical compartment. We may eventually see invasive infection (infection which enters anatomical compartments which are normally sterile e.g. the blood and central nervous system).

Let’s now consider COVID-19. In general, in severe disease, we see a disease state in which infection and inflammation are occurring in a widespread manner, with virus now having been isolated in extrapulmonary tissues like the gut, brain, kidney, liver, and heart. Given that infection occurs predominantly via inhalation, the fact that virus can be isolated from these tissues and there is associated immunopathology, suggests that the virus is able to effectively disseminate throughout the host despite attempts by the immune system to control it (of course, at least some of this immunopathology, possibly most, is the result of cytokine-mediated effects, but the root of this appears to be an a priori failure to control viral load). What’s more is that in some patients, a major feature of the maladaptive immune response is a failure to control viral replication wherein a persistently elevated viral load continues to stimulate a pro-inflammatory response that is associated with poor outcomes. Taken together, one can reasonably infer that a successful immune response against SARS-CoV-2 requires a very rapid induction of antiviral defenses in the early stages of infection, and furthermore, the fact that high levels of neutralizing antibodies seem to be emerging as a correlate of protection, suggests that the fact that these vaccines demonstrate efficacy against milder cases of COVID-19 indicates that they will be effective against severe COVID-19 as well (and the severe infections in Pfizer and Moderna’s trials, though limited in number, support this).

What don’t we know yet?

Durability of the vaccine candidates remains a critical open question. There’s no way to know this without waiting long enough for vaccinated people to start getting sick again, unfortunately, but there is good reason to be optimistic on this count, as a recent study suggests that immunity to SARS-CoV-2 following infection is likely long-lived, even in spite of the complex antagonism of the immune system by SARS-CoV-2.

It is not clear whether or to what extent the vaccines are able to interrupt transmission of SARS-CoV-2. Put another way, these trials may suggest good efficacy but we don’t yet know about the effectiveness.

References

1. N. E. Dean, P.-S. Gsell, R. Brookmeyer, V. De Gruttola, C. A. Donnelly, M. E. Halloran, M. Jasseh, M. Nason, X. Riveros, C. H. Watson, A. M. Henao-Restrepo, I. M. Longini, Design of vaccine efficacy trials during public health emergencies. Sci. Transl. Med. 11, eaat0360 (2019).
2. DESIGN OF VACCINE EFFICACY TRIALS TO BE USED DURING PUBLIC HEALTH EMERGENCIES — POINTS OF CONSIDERATIONS AND KEY PRINCIPLES. Who.int.
3. Halloran M, Longini I, Struchiner C. Design and Analysis of Vaccine Studies. Springer Science+Business Media; 2010.
4. O’neill R. On sample sizes to estimate the protective efficacy of a vaccine. Statistics in Medicine. 1988;7(12):1279–1288.
5. Pfizer and BioNTech Conclude Phase 3 Study of COVID-19 Vaccine Candidate, Meeting All Primary Efficacy Endpoints | Pfizer. Pfizer.com. 2020.
6. Analyzing Phase 3 COVID-19 Vaccine Trials | School of Public Health | University of Illinois at Chicago. Publichealth.uic.edu. 2020.
7. Dan J, Mateus J, Kato Y, Hastie K, Faliti C, Ramirez S, Frazier A, Yu E, Grifoni A, Rawlings S et al. Immunological memory to SARS-CoV-2 assessed for greater than six months after infection. biorxiv. 2020.
8. Chu H, Halloran M. Bayesian estimation of vaccine efficacy. Clinical Trials. 2004;1:306–314.
9. Gilbert P, Janes H, Huang Y. Power/Sample Size Calculations for Assessing Correlates of Risk in Clinical Efficacy Trials. Stat Med. 2016;35(21):3745–3759.
10. Moderna’s COVID-19 Vaccine Candidate Meets its Primary Efficacy Endpoint in the First Interim Analysis of the Phase 3 COVE Study | Moderna, Inc. Moderna, Inc. 2020.
11. A Phase 3, Randomized, Stratified, Observer-Blind, Placebo-Controlled Study to Evaluate the Efficacy, Safety, and Immunogenicity of mRNA-1273 SARS-CoV-2 Vaccine in Adults Aged 18 Years and Older. Modernatx.com. 2020.
12. A PHASE 1/2/3, PLACEBO-CONTROLLED, RANDOMIZED, OBSERVER-BLIND, DOSE-FINDING STUDY TO EVALUATE THE SAFETY, TOLERABILITY, IMMUNOGENICITY, AND EFFICACY OF SARS-COV-2 RNA VACCINE CANDIDATES AGAINST COVID-19 IN HEALTHY INDIVIDUALS. Pfe-pfizercom-d8-prod.s3.amazonaws.com. 2020.
13. Kucharski A, Eggo R, Watson C, Camacho A, Funk S, Edmunds W. Effectiveness of Ring Vaccination as Control Strategy for Ebola Virus Disease. Emerging Infectious Diseases. 2016;22(1):105–108.
14. B. D. Kirkpatrick, S. S. Whitehead, K. K. Pierce, C. M. Tibery, P. L. Grier, N. A. Hynes, C. J. Larsson, B. P. Sabundayo, K. R. Talaat, A. Janiak, M. P. Carmolli, C. J. Luke, S. A. Diehl, A. P. Durbin, The live attenuated dengue vaccine TV003 elicits complete protection against dengue in a human challenge model. Sci. Transl. Med. 8, 330ra36 (2016).
15. Human Challenge Trials for Vaccine Development: regulatory considerations. Who.int. 2016.
16. Will a small, long-shot U.S. company end up producing the best coronavirus vaccine?. 2020 [accessed 2020 Nov 19]. https://www.sciencemag.org/news/2020/11/will-small-long-shot-us-company-end-producing-best-coronavirus-vaccine
17. Pinkbook | Varicella | Epidemiology of Vaccine Preventable Diseases | CDC. Cdc.gov. 2020.
18. Guidelines on clinical evaluation of vaccines: regulatory expectations. Who.int. 2016.
19. Nauta J. Statistics in Clinical Vaccine Trials. Springer-Verlag Berlin Heidelberg; 2011.
20. Clemens J. Evaluating new vaccines for developing countries. Efficacy or effectiveness?. JAMA: The Journal of the American Medical Association. 1996;275(5):390–397.
21. Statistical Considerations for Clinical Trials During the COVID-19 Public Health Emergency Guidance for Industry. Fda.gov. 2020.
22. Development and Licensure of Vaccines to Prevent COVID-19 Guidance for Industry. Fda.gov. 2020.
23. Halloran M, Longini I, Struchiner C. Design and Interpretation of Vaccine Field Studies. Epidemiologic Reviews. 1999;21(1):73–88.
24. Agbabiaka, T.B., Savović, J. & Ernst, E. Methods for Causality Assessment of Adverse Drug Reactions. Drug-Safety 31, 21–37 (2008). https://doi.org/10.2165/00002018-200831010-00003
25. Iannelli MD V. How Often Should You Do Vaccine Titer Testing? — VAXOPEDIA. VAXOPEDIA. 2019.
26. Tokars J, Patel M, Foppa I, Reed C, Fry A, Ferdinands J. Waning of Measured Influenza Vaccine Effectiveness Over Time: The Potential Contribution of Leaky Vaccine Effect. Clinical Infectious Diseases. 2020.
27. Human Challenge Trials for Vaccine Development: regulatory considerations. Who.int. 2016.
28. Keech C, Albert G, Cho I, Robertson A, Reed P, Neal S, Plested J, Zhu M, Cloney-Clark S, Zhou H et al. Phase 1–2 Trial of a SARS-CoV-2 Recombinant Spike Protein Nanoparticle Vaccine. New England Journal of Medicine. 2020.
29. Toxicity Grading Scale for Healthy Adult and Adolescent Volunteers Enrolled in Preventive Vaccine Clinical Trials. Fda.gov. 2020.
30. Benefits of Influenza Vaccination: Selected Publications | CDC. Cdc.gov. 2016.
31. Addetia A, Crawford K, Dingens A, Zhu H, Roychoudhury P, Huang M, Jerome K, Bloom J, Greninger A. Neutralizing Antibodies Correlate with Protection from SARS-CoV-2 in Humans during a Fishery Vessel Outbreak with a High Attack Rate. Journal of Clinical Microbiology. 2020;58(11).
32. Gupta, A., Madhavan, M.V., Sehgal, K. et al. Extrapulmonary manifestations of COVID-19. Nat Med 26, 1017–1032 (2020). https://doi.org/10.1038/s41591-020-0968-3
33. Fajnzylber, J., Regan, J., Coxen, K. et al. SARS-CoV-2 viral load is associated with increased disease severity and mortality. Nat Commun 11, 5493 (2020). https://doi.org/10.1038/s41467-020-19057-5
34. Lindner D, Fitzek A, Bräuninger H, Aleshcheva G, Edler C, Meissner K, Scherschel K, Kirchhof P, Escher F, Schultheiss H et al. Association of Cardiac Infection With SARS-CoV-2 in Confirmed COVID-19 Autopsy Cases. JAMA Cardiology. 2020;5(11):1281.
35. Puelles V, Lütgehetmann M, Lindenmeyer M, Sperhake J, Wong M, Allweiss L, Chilla S, Heinemann A, Wanner N, Liu S et al. Multiorgan and Renal Tropism of SARS-CoV-2. New England Journal of Medicine. 2020;383(6):590–592.
36. Neuroinvasion of SARS-CoV-2 in human and mouse brain. REGENHEALTHSOLUTIONS (RHS). 2020.