The infection fatality rate (IFR) of COVID-19 is one of the most important parameters for mathematical models of the pandemic, yet it remains largely a mystery because we don’t yet know how many people have actually been infected. The IFR is the number of people who die from COVID-19 divided by the number of people who get infected. When you see the number of reported cases each day in the news, these reported cases include only people who have been diagnosed as having COVID-19 disease, or who have tested positive, but it does not include people who never get tested or are asymptomatic. Therefore, studies to assess the total number of infections are super important. This morning the results of a large-scale Stanford University study that tested for the presence of antibodies in the population of Santa Clara County, California were released.
- , , , , , , , , , , , , , , , ,
When conducting a large-scale community antibody test like this, the most important aspect of the experimental design is to make sure that the people selected for testing don’t self-select for the study. People who have reason to believe that they may have had COVID-19, or may have been exposed, will be vastly more eager to get tested than people who have no reason to think they may have had it. Ideally, you would either test everyone, or you should randomly select the participants without simply asking people to participate. If people self-select, this self-selection bias will result in a population of test subjects who are far more likely to have had the disease than the general population.
Today’s study is a very important one, and the researchers did a lot of other things right, but unfortunately they let participants self-select, so I believe that their reported results have overestimated the true prevalence in Santa Clara County. In this posting, I examine how to adjust their results to account for the self-selection bias.
How participants were selected
The following description of how participants were recruited is taken directly from their paper:
We recruited participants by placing targeted advertisements on Facebook aimed at residents of Santa Clara County. We used Facebook to quickly reach a large number of county residents and because it allows for granular targeting by zip code and sociodemographic characteristics. We used a combination of two targeting strategies: ads aimed at a representative population of the county by zip code, and specially targeted ads to balance our sample for under-represented zip codes. In addition, we capped registration from overrepresented areas. Individuals who clicked on the advertisement were directed to a survey hosted by the Stanford REDcap platform, which provided information about the study.
Basically, they put out a targeted ad on Facebook. Although it doesn’t say so here in the paper, I have heard second hand from people here in Santa Clara county that the ad offered a $10 Amazon gift card to participate. They targeted the ad to a subset of the population in each zip code.
They made substantial efforts to obtain a representative sample from the different zip codes and demographic groups within the county, and then after the proportions of participants didn’t quite match these distributions, they adjusted their results for this aspect of sampling bias. Their attention to this aspect of sampling bias was good. But they made no adjustments for self-selection bias.
All subjects’ blood were drawn for testing on 3-Apr-2020 or 4-Apr-2020 in three drive-through test sites in Los Gatos, San Jose and Mountain View.
The study’s results
After eliminating some subjects for various technical reasons, the study ended up with 3,300 people with test results. Of those, 50 were positive test results, constituting 1.50% of the tests. After adjusting for demographic and geographic sampling biases, they adjusted this to an estimate of 2.81% positive. They then adjusted for the accuracy of the test (the actual accuracy is uncertain) to come up with a final estimate of prevalence between 2.49% to 4.16% percent, where the range is due to assumptions about the test’s accuracy. This prevalence translates to between 48,000 and 81,000 people in the county, which is 50-85 times the number of confirmed cases. This led to an IFR of 0.12% to 0.2%.
The results I just listed are directly from the paper. None of these results are adjusted for self-selection bias.
Self-selection adjustment
Suppose that people who have previously had COVID-19 are 10 times more likely to sign up for a test than people who have never been infected. This is called the likelihood ratio, which I’ll denote as L. In this example, L=10.
Let C denote whether a person has had COVID: C=true if they have, C=false if they haven’t. Let T denote whether that person enrolls in the test: T=true means they enroll, T=false means they don’t. With this notation, Bayes’ rule can be written as
![\[ odds( C | T ) = odds( C ) \cdot L \]](https://lumina.com/wp-content/ql-cache/quicklatex.com-83a7f1eb9b8dfd1d45d006d3fb70786d_l3.png "Rendered by QuickLaTeX.com")
Where 
is the odds a person had COVID-19 given that they take the test, and 
is the odds a person from the general population had COVID-19. The study determined 
, whereas what we care about is , which doesn’t have the self-selection sampling bias.
The study estimated the prevalence, p, which is related to odds as 
. With some simple algebra, the true prevalence is obtained by multiplying the study’s estimate by an adjustment, 
, given by
![\[ \alpha = {1 \over { L (1-p) + p }} \]](https://lumina.com/wp-content/ql-cache/quicklatex.com-d4d5d52a6e92f3eb889e878b8866f217_l3.png "Rendered by QuickLaTeX.com")
This adjustment is shown in the following graph as a function of the likelihood ratio.
.To adjust for self-selection bias, you need to estimate how many times more likely a person who had previously had COVID-19 would be to participate in the study, compared to a person who has never had it. This is the likelihood ratio. If you think that person would be 2.5 times as likely, use the self-selection adjustment for L=2.5 from the above graph, which is 
, and then multiply the study’s prevalence number by 0.4, which yields a prevalence between 1.00% and 1.69%. You can also multiply by to adjust the total number of infected people, which for L=2.5 is between 19,500 and 33,000. To adjust the IFR estimate, divide by , which yields an adjusted IFR estimate between 0.29% and 0.49%. Adjusted values for these results are shown in the next four graphs as a function of likelihood ratio, the two lines showing the lower and upper estimates.
Estimating the Likelihood ratio
To adjust the study’s results, you have to estimate the likelihood ratio — how many times more likely someone previously infected would be to participate compared to someone never infected. Of course, this is also a big unknown.
Because the person considering whether to register doesn’t know whether she has COVID-19 or not, you might find the likelihood ratio to be overly abstract. It is easier to compare how much more likely it would be for someone who had symptoms at some point to participate in the study than someone who never had any symptoms. Of course, you might also want to consider the possibility that a person may be more likely to register because they know they had come into contact with someone else, or because their job makes them more vulnerable, etc. Because there are these other possibilities, I chose not to decompose the likelihood ratio into other estimations.
I live in Santa Clara County, and I was aware that the study was taking place. Many of my family’s friends were also aware that it was taking place, and we know of at least one person who tried very hard (unsuccessfully) to find the ad so that she could participate, because as she said, she had been sick and wanted to know if she had had COVID-19. Hence, based on my own experiences, my personal guess is L=5. But there is nothing magic about my guess, yours may differ.
Summary
The results from today’s study may lead some people to conclude that COVID-19 is no worse than the common flu. But this adjustment for self-selection bias show that this is not true. As the graphs above show, the prevalence drops off quickly when adjusted for even a small self-selection bias.
Understanding how many people in the population have had COVID-19 is an extremely important parameter for mathematical models of the pandemic. It is a critical piece of knowledge for determining how deadly a COVID-19 infection really is, for projecting hospital capacity and how many people may die, and for deciding when the economy can be reopened. Large scale tests of the population, like the Stanford study published today, help us estimate the prevalence of COVID-19 in the population.
When conducting a large-scale prevalence study, it is important to eliminate self-selection bias. When people are allowed to decide whether to participate in the study, those who have reason to believe that they are more likely to have had it will be more likely to participate. This may include people who had suspicious symptoms at some point, who know they were exposed at some point, or who work in situations that put them at greater risk of infection. In this article, I discussed how we can attempt to adjust for self-selection bias in the large-scale Santa Clara County study that was published this morning.
good point! have you had any reaction from the study authors?
Hi Paul. As far as I know, not yet.
I agree. Their methodology was deeply and unnecessarily flawed. You raised good motivators for self-selection: having felt ill, having been around someone who felt ill, being in a higher risk profession. Also, note the bias that is inherent in the method of using Facebook as the messenger with a brief period between posting on FB and the actual testing. We would need significant information on the other behaviors of people who use FB this frequently and whether they are more or less likely to have engaged in practices that would have put them at risk of acquiring the virus. Back… Read more »
Thank you for the reference to the obstetric patents study. I hadn’t seen that yet. Of course, the population of pregnant women would also have a ascertainment bias, since pregnant woman might have been more careful than other people, etc. But, like you, I have seen a lot of studies that have attempted to estimate the total number of cases in various populations, to get IFRs, and I have the same impression that the SC County study’s result was a far outlier. Your characterization as failing the back-of-the-envelope smell test is a good characterization.
The obstetric patient study is really interesting. I think the more revealing statistic there was the identification of 22 completely asymptomatic women, with only 7 showing symptoms (4 plus 3 who subsequently developed fever, though some may have been postpartum endometriosis). This suggests that only 25% of infected individuals may exhibit symptoms. Of course, the immune systems of pregnant women at term are “interesting” (e.g. activated CD69+ NK cells are elevated, for example), and they tend to be young and fit (I wouldn’t consider pregnancy to be a co-morbidity). I would therefore be surprised if that 3:1 ratio of asymptomatic… Read more »
I’ve been getting a sense from studies like this one, the Runyun Li et al Science article on China asymptomatics, the Vo study, the Kenji Mizumotol Diamond Princess article, the Kimball et all CDC report for King Co. Wash. nursing facility, the Boston homeless shelter study, and others, that the percentage of the those get it but never develop symptoms is somewhere around 30%. There seems to be a fair amount of consistency emerging.
Dr. Chrisman, Thank you for working through a potential correction to the selection bias present in Dr. Bhattacharya’s study. This model illustrates the impact that volunteer selection bias could have on the study result. As you suggest, your assumption of L=5-10 is fairly arbitrary and does not necessarily aid in demonstrating a more accurate prevalence estimate. The survey data from Dr. Bhattacharya’s study that reported previous symptoms of volunteers could be compared to Google’s symptom tracker for the county. This would provide a fairly accurate way to adjust for volunteer bias. I would be interested in seeing the effect that… Read more »
Grace – very good point. They said they collected information participants’ symptoms, but they didn’t include that in their report. I think you are right that it would be useful for narrowing the actual amount of bias. Also, if we knew how many people actually saw the ads, versus how many signed up, it could help.
Lonnie, I believe you raised an excellent point regarding the bias.
However something bothers me in the first equation: L is defined as P(T|C)/P(T|C=false)
However Bayes:
P(C|T) = P(C)* P(T|C)/P(T)
So some redefinition of L may be needed here.
Yigal — Thanks for the feedback. Bayes’ rule can be written in terms of probabilities (as you did with P(C|T) = P(C)* P(T|C)/P(T)), or in terms of odds, as I did in the first equation. I used the odds form because I thought the likelihood ratio was a natural way to capture the relative likelihood of the two groups participating, and it already appears in odds form of Bayes’ rule. The odds a:b is the same as the probability a/(a+b), so it is possible to write it out in the probability form, using L = P(T|C) / P(T|not C).
Hi Lonnie, Spot on analysis, and looks like you might have extracted a more plausible number from the Bendavid et al data. I would also note that the serological test measures anyone who has ever been infected (i.e. in the last 120 days or so; though it can plausibly inform about recent infections [IgM] versus historic infections [IgG]), whereas the PCR test measures a snapshot of those that are currently infected. One would need to do some more fancy math(s) here, but this would also factor in to a better assessment of the real mortality rate. Remember that at least… Read more »
great work…much appreciated
Chris excellent analysis! Thanks
Thank you very much for the thoughtful analysis regarding self-selection bias! I would only add that the tendency to self-select seems to me very likely to relate not only to (a) those who experienced COVID-19 like symptoms (perhaps those who had a bout with un-diagnosed influenza or something else) and (b) a known contact with a person known to be infected, but also those who (c) have frequented those places associated with known cases and/or (d) know someone who was infected even if they had no direct contact with such person since the outbreak began. These latter might be more… Read more »
Great analysis Lonnie. I too had the sense that doubling or tripling prevalence estimates based upon demographic data when such a small number tested positive is likely to significantly compound any errors generated by self-selection as opposed to randomized selection of participants. It was pointed out to me that because asymptomatic cases can be a significant portion of positive cases, this fact may ameliorate some of the self selection bias. I do not know how to integrate that factor into the adjustments the study team made. Another important point, accuracy of the anti-body tests has been called into question and… Read more »
The possibility that these authors had an agenda, that motivated them to achieve high prevalence rates and low IFR, is disturbing. But I think you are right that there are reasons to worry that this happened with this Santa Clara County study. In addition to Dr. Bendavid, John Ioannidis is also an author, who has published a few high profile op eds making similar arguments that the isolation measures & resulting economic shutdown where unwarranted because we don’t know enough to rule out the possibility that it might be not that dangerous. I am not in a position to make… Read more »
The fact that study leaders Drs. Bendavid & Bhattacharya did not reveal their previous analysis of the Vo, Italy widespread test results stating infection prevalence was so high in Italy (3%) that CFR was only .06% in the discussion section of the SCC study indicates they did not want this fact revealed prior to publication. Drs. Bendavid & Bhattacharya also posited in the March 24 WSJ Op-Ed piece titled “Is Covid-19 as Deadly as They Say? that the total mortality of CV-19 in this country would be between 20K &40k, the high end be eclipsed in this country today. Study… Read more »
I have to say that I am embarrassed to have made the above comment. I’ve thought about this more, and I don’t think there was any biasing agenda in the paper. I think the authors attempted an honest experiment, but had several methodological problems with the study’s design.
A wise self reflection. Calling out technical challenges in producing high integrity sampling is fair. Imputing perverse motivations on its authors due to a few speculative accusations is amateur. As it stands, the Santa Clara study has provided valuable insight at a time when any sense for the true progression of the virus is needed to support policymaking.
While I fully understand why you corrected yourself and applaud you for that, I unfortunately had the opposite conclusion. I initially thought the researchers were well-intentioned but poorly trained scientists who were a bit too eager to publish their results. But the same group did a similar study in LA county and went on a press tour. And their preprint could only be found in redstate.com, a far right website. (you will understand what I mean by far right when you go there). There is a somewhat innocuous explanation for this. (they released the preprint to hundreds of media websites… Read more »
I don’t have a PhD after my name but I do find it kind of a headscratcher that a criticism of the accuracy of a paper includes an estimate based primarily on “Many of my family’s friends were also aware that it was taking place, and we know of at least one person who tried very hard (unsuccessfully) to find the ad so that she could participate, because as she said, she had been sick and wanted to know if she had had COVID-19”. You’re right that your guess isn’t magical but it is pretty comical. Considering the universal hype… Read more »
Brian — I accept your criticism. I work very hard to remain objective and avoid biases. I absolutely recognize, as you say, that we all have our own biases, and that I am no different. So when you let me know that I may be succumbing to a bias, I listen and I study that. The alpha multiplier in my article works in both directions. If you do want to explore how the numbers change if the self-selection bias were to be in the other direction, this would correspond to L<1, and you can apply the same equation. Even a… Read more »
HI Brian.. From personal experience I totally agree with your assessment that self selection may have gone the other direction. In Alberta where testing is now available, people are asked to register for testing based on a self assessment that includes runny nose and coughing.. for those who are freaked out, any sniffle is Covid. This might be just my availability bias but in my view there is equal likelihood that the paranoid signed up in droves.
Lonnie, very nice analysis. To be fair, Bendavid and his co-authors conceded that self-selection might be a problem: they wrote. “Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible. The overall effect of such biases is hard to ascertain.” So yes, you’re right, the study is not methodologically rigorous when it comes to recruitment strategy, but the bias can work both ways. Perhaps the results actually undercount the percentage of people infected, because it drew a bunch of uninfected… Read more »
Keep in mind that an IFR of 0.3% to 0.5% translates to about 660K to 1.1M deaths in the US if you just let if run its course. That isn’t counting the colateral deaths (due to lack of access to ICU facilities), which may be a bit larger than these numbers. In addition to IFR, the other factor is how many people would get the disease, and because there is no innate immunity in the population to COVID-19 as there is with the flu, there will be more cases. A quick way of estimating how many cases there would be… Read more »
Oops!! Forgot the extra zero! That will teach me not to do arithmetic late at night. I meant 0.030% or around 100K+ deaths — including a pre-vaccine resurgence later in the year (though maybe it will be like H1N1, where we got a vaccine in 9 months). We’ll probably be south of 60K in this phase. So it will be worse than H1N1, but not 1918. And again, we’d have to stratify that CFR by age and age/co-morbidity combinations. For someone over 70 with COPD, your chances may be 50-50 if you catch it. The other thing I wanted to… Read more »
It would be great it the IFR was 0.03%, but the evidence it is not that low is pretty overwhelming. As you say, if it were that low, the number of deaths in the US would end up around 60K-100K in the absence of any protective measures (I’d say 66K is a good expectation). We are almost into that range now WITH extreme physical isolation measures and nowhere near herd immunity. Even the Bendavid study concluded that the percentage of people infected in Santa Clara County was <4.5%, a county that even one expects should have one of the highest… Read more »
Thanks for that thoughtful reply, I will look at all of those. If you’re right, then we are damned if we do and damned if we don’t. The virus has to run its course, now, or later, but eventually. Who will win the policy decision then? If it’s Zeke Emmanuel, with his “it’s going to take 18 months,” at one extreme, we will destroy economic society as we know it. And even if we delay, say, another two months, and then open up, the consequences may still be catastrophic from BOTH a societal/economic AND and epidemiological point of view. Or,… Read more »
I know, it really stinks. I’m so sick of the economic impacts of the lock down, and would really love to be through this. The more I have studied it, the less I see it as a trade-off between economy and # of deaths, because the same things we need to do to return to a functional economy are the same things we need to do to squash the number of cases. I analyzed this in depth in my very long Triangle Suppression article: https://lumina.com/forecast-update-us-deaths-from-covid-19-coronavirus-in-2020/ My colleague, Max Henrion, has recently also written an article, much shorter, with more focus… Read more »
It’s possible to be sensitive to that colliding venn diagram, where you’re concerned about suppressing COVID-19, worried about the economic impacts, and concerned about government intervention/suppression/imposition. However, if we don’t suppress (r<<1) significantly first, what are the consequences of reopening, and seeing a large second wave of infections. Most of us on the epidemiological side of things are concerned about that second wave, especially in the more rural areas that haven't yet spiked. I fully anticipate we'll see areas reexert high case loads, and I wouldn't be surprised to see new epicenters of infection. What will be devastating is if… Read more »