Background: I was reading the CNN Corona virus live stream today (06-Oct-2020), and read the following:
"Coronavirus testing in the US is still lagging, but Fauci says we're better off than we were a few months ago", from CNN’s Health Shelby Lin Erdman
The coronavirus pandemic has brought countries to a standstill. In many places, as countries reopen, Covid-19 cases are on the rise. Follow here for the latest.
www.cnn.com
The article continues:
The Abbott test, which received an Emergency Use Authorization from the US Food and Drug Administration in August, returns test results in 15 minutes. The company says the test is 97% accurate in detecting positives and 98.5% accurately identifying people who are not infected. Abbott had trouble with accuracy in one of its previous coronavirus tests.
I was interested in how those accuracy numbers translate into false negative and false positive test results.
First, there is a question of the meaning the "accuracy" percentages. I found this, which indicates that the numbers are likely "positive percentage agreement" (PPA) and "negative percentage agreement" (NPA) values, since the numbers very closely match this published chart of test results:
"Abbott rides to the rescue with $5 Covid-19 test"
Read Evaluate's latest blogs, reports, webinars and thought leadership for detailed analysis of the state of the biopharma market.
www.evaluate.com
"Abbott reports positive and negative percent agreement"
"Accuracy of FDA-authorised antigen tests"
Date of EUA | Company | Test | PPA | NPA | Size* |
---|
Aug 26 | Abbott | BinaxNow | 97.1% | 98.5% | 102 |
Aug 18 | LumiraDx | LumiraDx | 97.6% | 96.6% | 257 |
Jul 2 | Beckton Dickenson | BD Veritor | 84% | 100% | 226 |
May 8 | Quidel | Sofia Sars | 96.7% | 100% | 209 |
* Suspected positive sample size
To understand what "positive percentage agreement" and "negative percentage agreement" mean, I found the following:
"Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests"
And the answer is ... complicated. PPA and NPA are measure of how well a test results match a non-standard reference test. Without knowing the accuracy of the reference test, understanding the Abbott test results is difficult. If the reference test was 100% accurate, then the PPA and NPA are percentage measurements of sensitivity and specificity. But, if the reference test is less than 100% accurrate, the PPA and NPA may indicate a lesser or a greater accuracy. For example, the test which is being compared to the reference test might agree with the reference test where the reference test is accurate, and dis-agree where the references test is inaccurate.
The FDA has this to say:
Comparing a new test to a non-reference standard does not yield true performance. If the new test is better than the non-reference standard, the agreement will be poor. Alternatively, the agreement could be poor because the non-reference standard is fairly accurate and the new test is inaccurate. There is no statistical solution to determining which scenario is the true situation.
In any case, labeling PPA and NPA as "accuracy" seems quite incorrect.
As an aside, two other values which are described by the FDA text are "sensitivity" and "specificity":
- Sensitivity refers to how often the test is positive when the condition of interest is present
- Specificity refers to how often the test is negative when the condition of interest is absent
Although of only small value -- given the uncertainty of the meaning of the Abbot PPA and NPA values, I took the values as measuring "accuracy" and determined the proportion of false results for different virus distributions. That is, for populations with 90%, 50%, 10%, 1%, 0.1%, and 0.01% infected individuals:
Populations descriptions:
| Percentage | Proportion | Number of Persons |
---|
Uninfected | 10% | 1/10 | 100,000 |
Infected | 90% | 9/10 | 900,000 |
Total | 100% | 10/10 | 1,000,000 |
| | | |
Uninfected | 50% | 5/10 | 500,000 |
Infected | 50% | 5/10 | 500,000 |
Total | 100% | 10/10 | 1,000,000 |
| | | |
Uninfected | 90% | 9/10 | 900,000 |
Infected | 10% | 1/10 | 100,000 |
Total | 100% | 10/10 | 1,000,000 |
| | | |
Uninfected | 99% | 99/100 | 990,000 |
Infected | 1% | 1/100 | 10,000 |
Total | 100% | 100/100 | 1,000,000 |
| | | |
Uninfected | 99.9% | 999/1000 | 999,000 |
Infected | 0.1% | 1/1000 | 1,000 |
Total | 100.0% | 1000/1000 | 1,000,000 |
| | | |
Uninfected | 99.99% | 9999/10000 | 999,900 |
Infected | 0.01% | 1/10000 | 100 |
Total | 100.00% | 10000/10000 | 1,000,000 |
Working through the numbers, I obtained the following test accuracy results. Here "accuracy" means "Upon obtaining a test result, what is the chance that that result is correct?"
Unsurprisingly, as the proportion of the population which is infected drops, the number of false positives dominates the "infected" results, leading to increasingly worse accuracy.
Population Percent Infected | Percent True Uninfected Results | Percent True Infected Results |
---|
90% | 87.8% | 99.7% |
50% | 98.5% | 97.0% |
10% | 99.8% | 78.5% |
1% | 99.98% | 24.9% |
0.1% | 99.998% | 3.1% |
0.01% | 99.9998% | 0.327% |
Calculations:
Code:
Uninfected: 10% ( 1/10) 100,000 U 97,000 I 3,000
Infected: 90% ( 9/10) 900,000 U 13,500 I 886,500
Total: 100% (10/10) 1,000,000 U 110,500 I 889,500 (U 87.8% I 99.7%)
Uninfected: 50% ( 5/10) 500,000 U 485,000 I 15,000
Infected: 50% ( 5/10) 500,000 U 7k500 I 492,500
Total: 100% (10/10) 1,000,000 U 492k500 I 507,500 (U 98.5% I 97.0%)
Uninfected: 90% ( 9/10) 900,000 U 873,000 I 27,000
Infected: 10% ( 1/10) 100,000 U 1,500 I 98,500
Total: 100% (10/10) 1,000,000 U 874,500 I 125,500 (U 99.8% I 78.5%)
Uninfected: 99% ( 99/100) 990,000 U 960,300 I 29,700
Infected: 1% ( 1/100) 10,000 U 150 I 9,850
Total: 100% (100/100) 1,000,000 U 960,450 I 39,550 (U 99.98% I 24.9%)
Uninfected: 99.9% ( 999/1000) 999,000 U 969,030 I 29,970
Infected: 0.1% ( 1/1000) 1,000 U 15 I 985
Total: 100.0% (1000/1000) 1,000,000 U 969,045 I 30,955 (U 99.998% I 3.1%)
Uninfected: 99.99% ( 9999/10000) 999,900 U 969,903.0 I 29,997.0
Infected: 0.01% ( 1/10000) 100 U 1.5 I 98.5
Total: 100.00% (10000/10000) 1,000,000 U 969,904.5 I 30,095.5 (U 99.9998% I 0.327%)
Any errors in the above are my own.
Tom Bitonti