standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| second reading | alternatives | an independent re-reading of a facet, recorded whether or not it agrees |
| relation to other work | claim_relations | how this paper stands to another: agrees, contradicts, extends, qualifies |
| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| content tag | content_tags | a finding, definition, quantity, statistic or variable named in the paper |
| contribution | contribution | what the paper claims to add |
| control | controls | an experiment run to eliminate a rival explanation |
| limitation | limitations | what the paper says its evidence cannot support |
| mechanism | mechanisms | how the reported effect is said to come about |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
| unrecordable relation | unhoused_relations | a relation the paper asserts that this vocabulary cannot express, with the reason |
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
Lancet Infect Dis Lancet Infect Dis 445 elsevierwt 101130150 The Lancet. Infectious Diseases 1473-3099 1474-4457 pmc-is-collection-domain yes pmc-collection-title Elsevier Sponsored Documents PMC7511171 PMC7511171.2 7511171 7689287 32979318 10.1016/S1473-3099(20)30634-4 S1473-3099(20)30634-4 2 Articles Performance characteristics of five immunoassays for SARS-CoV-2: a head-to-head benchmark comparison The National SARS-CoV-2 Serology Assay Evaluation Group Ainsworth Mark Andersson Monique Auckland Kathryn Baillie J Kenneth Barnes Eleanor Beer Sally Beveridge Amy Bibi Sagida Blackwell Luke Borak Martyna Bown Abbie Brooks Tim Burgess-Brown Nicola A Camara Susana Catton Matthew Chau Kevin K. Christott Thomas Clutterbuck Elizabeth Coker Jesse Cornall Richard J Cox Stuart Crawford-Jones David Crook Derrick W D'Arcangelo Silvia Dejnirattsai Wanwisa Dequaire Julie M M Dimitriadis Stavros Dingle Kate E Doherty George Dold Christina Dong Tao Dunachie Susanna J Ebner Daniel Emmenegger Marc Espinosa Alexis Eyre David W Fairhead Rory Fassih Shayan Feehily Conor Felle Sally Fernandez-Cid Alejandra Fernandez Mendoza Maria Foord Thomas H Fordwoh Thomas Fox McKee Deborah Frater John Gallardo Sanchez Veronica Gent Nick Georgiou Dominique Groves Christopher J Hallis Bassam Hammond Peter M Hatch Stephanie B. Harvala Heli J Hill Jennifer Hoosdally Sarah J Horsington Bryn Howarth Alison James Tim Jeffery Katie Jones Elizabeth Justice Anita Karpe Fredrik Kavanagh James Kim David S Kirton Richard Klenerman Paul Knight Julian C Koukouflis Leonidas Kwok Andrew Leuschner Ullrich Levin Robert Linder Aline Lockett Teresa Lumley Sheila F Marinou Spyridoula Marsden Brian D Martinez Jose Martins Ferreira Lucas Mason Lara Matthews Philippa C Mentzer Alexander J Mobbs Alexander Mongkolsapaya Juthathip Morrow Jordan Mukhopadhyay Shubhashish M M Neville Matthew J Oakley Sarah Oliveira Marta Otter Ashley Paddon Kevin Pascoe Jordan Peng Yanchun Perez Elena Perumal Prem K Peto Timothy E A Pickford Hayleah Ploeg Rutger J Pollard Andrew J Richardson Anastasia Ritter Thomas G Roberts David J Rodger Gillian Rollier Christine S Rowe Cathy Rudkin Justine K Screaton Gavin Semple Malcolm G Sienkiewicz Alex Silva-Reyes Laura Skelly Donal T Sobrino Diaz Alberto Stafford Lizzie Stockdale Lisa Stoesser Nicole nicole.stoesser@ndm.ox.ac.uk * Street Teresa Stuart David I Sweed Angela Taylor Adan Thraves Hannah Tsang Hoi P Verheul Marije K Vipond Richard Walker Timothy M Wareing Susan Warren Yolanda Wells Charlie Wilson Clare Withycombe Kate Young Rebecca K † * Correspondence to: Dr Nicole Stoesser, Department of Microbiology, John Radcliffe Hospital, Headington, Oxford OX3 9DU, UK nicole.stoesser@ndm.ox.ac.uk † Group members listed in the appendix (pp 1–5)
12 2020 20 12 369547 1390 1400 01 12 2020 24 09 2020 17 02 2026 7511171 10.1016/S1473-3099(20)30634-4 1 23 09 2020 7689287 10.1016/S1473-3099(20)30634-4 2 01 12 2020 © 2020 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license 2020 https://creativecommons.org/licenses/by/4.0/ This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Correction to Lancet Infect Dis 2020; 20: 1390–400 20 12 25 11 2020 e298 e298 The Lancet. Infectious Diseases PMC7709078 33248043 The complexities of SARS-CoV-2 serology 20 12 23 9 2020 1350 1351 The Lancet. Infectious Diseases 10.1016/S1473-3099(20)30699-X PMC7511169 32979317 Summary
Background
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗sensesconcept_edgesSevere acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has caused a global pandemic in 2020. Testing is crucial for mitigating public health and economic effects. Serology is considered key to population-level surveillance and potentially individual-level risk assessment. research_questionalternativesHowever, immunoassay performance has not been compared on large, identical sample sets. alternativesWe aimed to investigate the performance of four high-throughput commercial SARS-CoV-2 antibody immunoassays and a novel 384-well ELISA.
Methods
alternativesWe did a head-to-head assessment of SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA), LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy), Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland), SARS-CoV-2 Total assay (Siemens, Munich, Germany), and a novel 384-well ELISA (the Oxford immunoassay). We derived sensitivity and specificity from 976 pre-pandemic blood samples (collected between Sept 4, 2014, and Oct 4, 2016) and 536 blood samples from patients with laboratory-confirmed SARS-CoV-2 infection, collected at least 20 days post symptom onset (collected between Feb 1, 2020, and May 31, 2020). Receiver operating characteristic (ROC) curves were used to assess assay thresholds.
Findings
object_of_studyclaim_typecontrolsclaim_relationscontent_tags↗At the manufacturers' thresholds, for the Abbott assay sensitivity was 92·7% (95% CI 90·2–94·8) and specificity was 99·9% (99·4–100%); for the DiaSorin assay sensitivity was 96·2% (94·2–97·7) and specificity was 98·9% (98·0–99·4); for the Oxford immunoassay sensitivity was 99·1% (97·8–99·7) and specificity was 99·0% (98·1–99·5); for the Roche assay sensitivity was 97·2% (95·4–98·4) and specificity was 99·8% (99·3–100); and for the Siemens assay sensitivity was 98·1% (96·6–99·1) and specificity was 99·9% (99·4–100%). method↗mechanismssensescontent_tagscontent_tagsAll assays achieved a sensitivity of at least 98% with thresholds optimised to achieve a specificity of at least 98% on samples taken 30 days or more post symptom onset.
Interpretation
Four commercial, widely available assays and a scalable 384-well ELISA can be used for SARS-CoV-2 serological testing to achieve sensitivity and specificity of at least 98%. The Siemens assay and Oxford immunoassay achieved these metrics without further optimisation. This benchmark study in immunoassay assessment should enable refinements of testing strategies and the best use of serological testing resource to benefit individuals and population health.
Funding
Public Health England and UK National Institute for Health Research.
pmc-status-qastatus 0 pmc-status-live yes pmc-status-embargo no pmc-status-released yes pmc-prop-open-access yes pmc-prop-olf no pmc-prop-manuscript no pmc-prop-legally-suppressed no pmc-prop-has-pdf no pmc-prop-has-supplement yes pmc-prop-pdf-only no pmc-prop-suppress-copyright no pmc-prop-is-real-version no pmc-prop-is-scanned-article no pmc-prop-preprint no pmc-prop-in-epmc yes pmc-license-ref CC BY Introduction
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationsconcept_edgesSevere acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has emerged as a novel human pathogen, causing a global pandemic in 2020, with more than 25 million confirmed infections and more than 840 000 deaths to date. 1 Testing and case ascertainment have been crucial to controlling virus transmission and in developing public health and political strategies to mitigate the effect of this pathogen.
SARS-CoV-2 testing broadly takes two forms: first, direct detection of the virus in respiratory samples with real-time RT-PCR; and second, by using serology to investigate the presence of antibodies. 2 Immunoassays detect either specific types of antibody (eg, IgM or IgG) or total antibody. SARS-CoV-2 antibodies typically start to appear at least 5–7 days post infection 3 and are therefore an unreliable marker for early acute infection. The degree and duration of immunity that antibodies confer are unclear. A prominent use for serological testing has therefore been at a population level, for informing the extent of population exposure. Other uses include assessing risk of infection at an individual level and to support research and development (eg, quantifying antibody responses in vaccine trials). 4
Several manufacturers have developed immunoassays compatible with global laboratory infrastructures, enabling widespread testing of hundreds to thousands of samples per day. However, the scale-up required for regular population-wide testing (eg, every few weeks or months) might exceed the capacity of commercial platforms. limitationscontent_tagsTo date, few thorough, direct assessments of immunoassay performance on large sample sets have been done, and governments, regulators, and clinical laboratories have had to balance the urgent need to facilitate the demand for serological testing with the few data available on assay performance, leading to a relaxation of typical assessment criteria (eg, the US Food and Drug Administration's [FDA] Emergency Use Authorization programme). 4
Research in context
Evidence before this study
concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsSubstantial global interest exists in the use of serology to enable population-level surveillance of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection and to inform individual-level management and risk stratification; however, it is unclear which widely available SARS-CoV-2 serological immunoassays perform to the standards required to meet these needs. The most widely used commercial assays that have obtained regulatory approvals for emergency use in the USA and the EU include the SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA), LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy), Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland), and the anti-SARS-COV-2 ELISA (IgG; EUROIMMUN, Lübeck, Germany). limitationsconcepts↗content_tagsHowever, the sensitivity of the EUROIMMUN assay has been reported as being approximately 90%. We therefore replaced this with the SARS-CoV-2 Total assay (Siemens, Munich, Germany) in a head-to-head comparison. In addition to reviewing performance assessments undertaken by each manufacturer, we searched PubMed, BioRxiv, and MedRxiv up to May 31, 2020, using the following search terms: (SARS-CoV-2) AND ([ELISA] OR [EIA] OR [CLIA] OR [FIA] OR [IFA] OR [IgG]). Details and expanded PubMed search terms are given in the appendix (pp 7, 8) . We also reviewed any investigations of relevant assays by Public Health England (PHE) up to May 31, 2020. Of 423 studies assessed, full-text reviews were done for 124 articles. Eight studies provided data on sensitivity and specificity for the Abbott (four studies) or DiaSorin (five) assays; PHE investigations were undertaken for the Abbott and Roche assays. mechanismsclaim_relationscontent_tagscontent_tagsFor the Abbott assay, sensitivity ranged from 93·4% to 100% for samples taken at least 14 days post symptom onset (70 to 680 samples) and specificity from 95·1% to 100% (nine to 1020); for the DiaSorin assay, we found a single sensitivity estimate of 94·4% on samples taken at least 14 days post symptom onset (18 samples) and specificity ranged from 94·9% to 100% (69 to 1140 samples). For the Roche assay, we found a single sensitivity estimate of 87·0% on samples taken at least 14 days post symptom onset (n=77) and a single specificity estimate of 100% (n=472). contributioncontrolscontent_tagsNo published studies assessed all four commercial platforms using the same sample sets.
Added value of this study
concepts↗concepts↗concepts↗concepts↗concepts↗sensesIn this study on a large, single sample set of more than 1500 samples, we compared four commercial SARS-CoV-2 immunoassays (Abbott, DiaSorin, Roche, and Siemens assays) with a global installed base, and a novel 384-well ELISA (the Oxford immunoassay) that could scale to be used for population-level surveillance. The Siemens assay and Oxford immunoassay achieved a sensitivity and specificity of at least 98% without optimisation (lower 95% CI ≥96%). The other assays could achieve sensitivities and specificities of at least 98% with assay threshold adjustment or deployment on samples taken at least 30 days after symptom onset. mechanismsconcepts↗concepts↗concept_edgescontent_tagscontent_tagscontent_tagsHowever, differences observed in assay performance translate into thousands of additional incorrect diagnoses between the worst and best platforms if millions of tests are done in large populations. For example, at a 10% seroprevalence, the Siemens assay would generate an estimated 2800 total errors per million tests, versus the DiaSorin assay at a rate of 13 700 total errors per million tests.
Implications of all the available evidence
Understanding the comparative performance of immunoassays is crucial to the development of appropriate individual and population-level SARS-CoV-2 testing strategies. We showed that although precise performance metrics varied between immunoassay platforms, all assays that we assessed could be usefully deployed with careful consideration of use case, assay thresholds, and by considering symptom-to-sample timings, thus optimising the use of available serological testing resource and enabling the most widespread rollout. We also showed that a novel 384-well format ELISA would potentially be scalable for population-level testing of hundreds of thousands of samples.
To facilitate national public health and health-care providers in choosing platforms that are appropriate for SARS-CoV-2 serological testing, we aimed to directly assess the performance of four commercial SARS-CoV-2 antibody immunoassays, with the primary aim of identifying the sensitivity and specificity of each assay. We also assessed the same samples using a novel 384-well format ELISA that targeted antibodies against trimeric spike protein (the Oxford immunoassay).
Methods
Study design
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗To date, the UK has one of the few established standards for performance metrics for SARS-CoV-2 immunoassays internationally; no specific guidance exists on performance metrics from either the US FDA or the European Centre for Disease Prevention and Control. concept_edgesconcept_edgesWe established a blood (plasma and serum) sample collection including pre-pandemic samples from healthy individuals aged at least 18 years, collected between Sept 4, 2014, and Oct 4, 2016, as part of the Oxford Biobank, 5 a population-based cohort of individuals in Oxfordshire, UK, who consented to participation in research studies, and samples from individuals aged at least 18 years with laboratory-confirmed SARS-CoV-2 infection from cohorts of patients admitted to hospital or surveillance on health-care workers ( appendix p 9 ). These samples enabled us to assess the immunoassays in line with the UK Medicines and Healthcare products Regulatory Agency (MHRA) target product profile for enzyme immunoassays, 6 requiring known negative samples to be taken more than 6 months before the known appearance of SARS-CoV-2 (ie, earlier than July 2019), and known positive samples to be collected from individuals with a previous positive SARS-CoV-2 RT-PCR from a nose or throat swab and at least 20 days post symptom onset ( appendix pp 13, 14 ). The sample collection also enabled us to assess immunoassay performance on samples taken within 20 days after symptom onset as part of secondary analyses. In each case samples were de-duplicated by individual, and the latest sample from each individual was analysed. Two assays (SARS-CoV-2 IgG assay [Abbott, Chicago, IL, USA] and LIAISON SARS-CoV-2 S1/S2 IgG [DiaSorin, Saluggia, Italy]) were done in Oxford, UK (clinical biochemistry and microbiology laboratories, John Radcliffe Hospital), and two (Elecsys Anti-SARS-CoV-2 assay [Roche, Basel, Switzerland] and SARS-CoV-2 Total assay [Siemens, Munich, Germany]) at Public Health England (PHE) Porton Down.
Additionally, we analysed an eight-point 1:2 dilution series of three known high-volume plasma samples obtained from SARS-CoV-2-positive patients recruited by the UK National Health Service Blood and Transplant (NHSBT; RT-PCR positive, at least 20 days post symptom onset) with high (donor ID 10062), medium (10061), and low (10063) titre SARS-CoV-2 IgG antibodies as assessed by the SARS-CoV-2 IgG ELISA (EUROIMMUN, Lübeck, Germany; ratio values of 33·33, 4·34, and 2·50, respectively; ratio is the optical density of the sample divided by the optical density of the calibrator [ratio ≥1·1 is positive]), and one known negative control (BD001). Samples were analysed at minimum in triplicate on the four commercial immunoassays, and singly on the Oxford immunoassay. The NHSBT samples were correlated with national SARS-CoV-2 reagents developed by the UK's National Institute for Biological Standards and Control (NIBSC; 20/130 [single donor, high-titre antibody], 20/120 [single donor, relatively high-titre antibody], 20/122 [pool of five donor samples, mid-titre antibody], 20/124 [low S1, high-nucleocapsid protein antibody titre], 20/126 [low-titre antibody], 20/128 [negative control] 7 , 8 ) on the Oxford immunoassay ( appendix p 10 ).
We drafted a protocol before implementing the study, which was shared with the manufacturers and the UK Department of Health and Social Care.
Procedures
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsThe installed base of analysers running the investigated assays is global, making their widespread deployment feasible. Two assays targeted the nucleocapsid protein (Abbott and Roche) and two had spike protein-based targets (DiaSorin and Siemens); two were IgG assays (Abbott and DiaSorin), and two were total antibody assays (Roche and Siemens). Manufacturers' sensitivity estimates ranged from 96·8% to 100% on samples taken at least 14 days post symptom onset or RT-PCR, and specificity estimates ranged from 98·5% to 99·8% on pre-pandemic samples ( appendix p 14 ). 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18
Assays were done in accordance with the manufacturers' instructions by trained laboratory staff in laboratories that were accredited by the UK Accreditation Service, on appropriate analysers, and with the specified controls and calibrants, using thresholds for calling positives and negatives set by the manufacturer for testing in the UK ( appendix p 14 ), but also reporting quantitative values. Samples were assigned unique study barcodes for this study, and laboratory staff doing the assays did not have access to sample metadata before running the assays.
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsconcept_edgesThe Oxford immunoassay is an indirect ELISA, measuring serum IgG against trimeric spike protein, 19 using a horseradish peroxidase-linked anti-human IgG antibody as the secondary. We implemented this assay as previously described, 20 with minor modifications ( appendix p 10 ). concept_edgesReadouts were measured as a fluorescent signal, normalised to standard units by calibrating to a dilution series of the NHSBT controls and a monoclonal antibody (CR3022), using a natural cubic spline-based regression model to enable across-plate and across-batch comparisons ( appendix p 10 ). We derived thresholds for the Oxford immunoassay using an independent set of serum samples from 44 individuals with acute SARS-CoV-2 infection; 99 convalescent individuals post SARS-CoV-2 infection; one Middle East respiratory syndrome coronavirus (MERS-CoV) antiserum; 23 individuals with other respiratory virus infections; and 1205 pre-pandemic samples ( appendix pp 10, 11 ). After at least 10 days post symptom onset, excluding the MERS-CoV antiserum (which was positive) but including the 23 samples from individuals with other viral infections, derivation sensitivity was 100% (120 of 120 [95% CI 97·0–100·0]) and specificity (at a normalised threshold of 8 million units) was 99·6% (1223 of 1228 [99·1–99·9).
The Oxford immunoassay runs batches of up to 12 384-well plates, each containing 3200 samples, over a period of 7 h to generate data on up to 3840 samples and controls per run. Samples and plates were handled using a Janus automated liquid handler (Perkin Elmer, Waltham, MA, USA), and plates were read with an EnVision 2104 multilabel plate reader (Perkin Elmer). The assay requires low input sample volumes of the order of 2 μL serum sample, subsequently diluted 1:25 in sample buffer (1% milk in phosphate-buffered saline with Tween); however, dead volume requirements of approximately 10 μL are required for adequate liquid handling function.
Sample aliquots of 0·5–2 mL were stored at −20°C before the study, and freeze-thaw cycles were minimised (fewer than five cycles). After thawing for processing, samples were spun for 10 min at 1844 g and the supernatant removed for subsequent work. No further freeze or thaw steps were taken during the study and samples were kept at 4°C for up to 7 days to complete all runs. For the Oxford immunoassay assessment, thawed samples were arrayed in 384-well plates and stored at −20°C with a single further thaw before processing.
Outcomes and statistical analysis
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗Whereas sample numbers were partly decided by the availability of samples required to address the study aims, the sample size was prospectively calculated to provide 80% power to detect a lower 95% CI of at least 96·0% if the actual assay sensitivity was 98·3% (which required ≥460 positive samples) and 80% power to detect a lower 95% CI of at least 96·0% if the actual assay specificity was 97·7% (which required ≥901 negative samples). These metrics were in line with the manufacturers' performance characteristics.
The data were collated, cleaned, and locked before analysis. Sensitivity and specificity with exact binomial 95% CIs for each assay were compared with current UK MHRA guidance 6 stipulating a requirement for sensitivity and specificity of at least 98%, with the lower bound of the 95% CI to be at least 96%. Analyses were carried out for samples for which results were available across all platforms as the primary analysis, and for all samples as part of secondary analyses. When more than one sample was available from the same individual, samples were de-duplicated by individual and only the last sample obtained was analysed, unless otherwise indicated.
Other secondary analyses were done; first, using receiver operator characteristic (ROC) curves to define trade-offs in assay sensitivity and specificity. Assay performance was also assessed according to the sampling timepoint (days post symptom onset and post RT-PCR test; at intervals of ≥14 days [per original protocol], ≥20 days [per MHRA], and ≥30 days). Evidence of differences in immunoassay sensitivity with respect to concept_edgessymptom severity (defined as asymptomatic, mild, severe, critical or death, in line with WHO criteria 21 ) for the subgroup of samples for which this information was available was assessed for each immunoassay. Positive and negative predictive values at population prevalences of 5%, 10%, 20%, and 50% previous SARS-CoV-2 infection were modelled. The percentage of positive tests from SARS-CoV-2-positive individuals confirmed by RT-PCR over time and by serology platform was assessed; for this analysis all samples were included. Trajectories of antibody titres with respect to days post symptom onset were modelled by fitting spline curves (three to five knots; model selection based on the Akaike Information Criterion); for this analysis the first sample per patient was included. Statistical analyses and data visualisations were done in R (version 3.6.3) and Stata/IC (version 16.1). Analyses and results were overseen by an external review group.
For the DiaSorin assay, for which the manufacturer specifies repeat testing in the event that results fall within an equivocal zone (12·0 ≤ × <15·0 AU/mL), we pre-specified that we were unable to do repeat testing because of insufficient sample volumes available. These samples were excluded from the primary sensitivity and specificity calculations for the DiaSorin assay, but subsequently included in the ROC analyses. STARD and PRISMA checklists are included in the appendix (pp 37–40) .
Role of the funding source
The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Results
concepts↗concepts↗From 1000 pre-pandemic samples (collected between Sept 4, 2014, and Oct 4, 2016), and 769 samples from 673 individuals with laboratory-confirmed SARS-CoV-2 infection (collected between Feb 1, and May 31, 2020), assay sensitivity and specificity were assessed on 976 known negative pre-pandemic samples and 536 samples from RT-PCR-confirmed COVID-19 patients, de-duplicated by individual for whom complete data across all platforms were available ( appendix p 24 ). No samples failed testing on any of the four commercial platforms. We found 18 liquid handling failures on the Oxford immunoassay resulting in empty wells. Sensitivity and specificity of each of the assays are shown in the table and figure 1 . Similar results were obtained analysing all available samples ( appendix p 27 ; table ). The clearest separation of known positive and known negative samples was shown for the Siemens assay ( figure 2 ; appendix p 27 [all samples]). Table Sensitivity, specificity, and sample sizes for all groups and subgroups assessed
Specificity Sensitivity
Detected Equivocal Not detected Not available Total Specificity (95% CI) Detected Equivocal Not detected Not available Total Sensitivity (95% CI)
Samples with complete data available
Abbott 1 .. 975 .. 976 99·9% (99·4–100) 497 .. 39 .. 536 92·7% (90·2–94·8)
DiaSorin 11 2 963 .. 976 98·9% (98·0–99·4) 509 7 20 .. 536 96·2% (94·2–97·7)
Oxford immunoassay 10 .. 966 .. 976 99·0% (98·1–99·5) 531 .. 5 .. 536 99·1% (97·8–99·7)
Roche 2 .. 974 .. 976 99·8% (99·3–100) 521 .. 15 .. 536 97·2% (95·4–98·4)
Siemens 1 .. 975 .. 976 99·9% (99·4–100) 526 .. 10 .. 536 98·1% (96·6–99·1)
Samples with complete data available (specificity set to ≥98%)
Abbott 19 .. 957 .. 976 98·1% (97·0–98·8) 523 .. 13 .. 536 97·6% (95·9–98·7)
DiaSorin 18 0 958 .. 976 98·2% (97·1–98·9) 523 0 13 .. 536 97·6% (95·9–98·7)
Oxford immunoassay 19 .. 957 .. 976 98·1% (97·0–98·8) 533 .. 3 .. 536 99·4% (98·4–99·9)
Roche 19 .. 957 .. 976 98·1% (97·0–98·8) 533 .. 3 .. 536 99·4% (98·4–99·9)
Siemens 15 .. 961 .. 976 98·5% (97·5–99·1) 530 .. 6 .. 536 98·9% (97·6–99·6)
All samples
Abbott 1 .. 994 .. 995 99·9% (99·4–100) 500 .. 40 .. 540 92·6% (90·0–94·7)
DiaSorin 12 2 981 .. 995 98·8% (97·9–99·4) 512 7 21 .. 540 96·1% (94·0–97·5)
Oxford immunoassay 10 .. 967 18 977 99·0% (98·1–99·5) 535 .. 5 .. 540 99·1% (97·9–99·7)
Roche 2 .. 993 .. 995 99·8% (99·3–100) 521 .. 15 4 536 97·2% (95·4–98·4)
Siemens 1 .. 993 1 994 99·9% (99·4–100) 526 .. 10 4 536 98·1% (96·6–99·1)
All samples (specificity set to ≥98%)
Abbott 19 .. 976 .. 995 98·1% (97·0–98·8) 527 0 13 .. 540 97·6% (95·9–98·7)
DiaSorin 19 0 976 .. 995 98·1% (97·0–98·8) 527 .. 13 .. 540 97·6% (95·9–98·7)
Oxford immunoassay 19 .. 958 18 977 98·1% (97·0–98·8) 537 .. 3 .. 540 99·4% (98·4–99·9)
Roche 19 .. 976 .. 995 98·1% (97·0–98·8) 533 .. 3 4 536 99·4% (98·4–99·9)
Siemens 15 .. 979 1 994 98·5% (97·5–99·2) 530 .. 6 4 536 98·9% (97·6–99·6)
Samples with complete data available taken ≥14 days post symptom onset
Abbott 1 .. 975 .. 976 99·9% (99·4–100) 520 .. 41 .. 561 92·7% (90·2–94·7)
DiaSorin 11 2 963 .. 976 98·9% (98·0–99·4) 529 8 24 .. 561 95·7% (93·6–97·2)
Oxford immunoassay 10 .. 966 .. 976 99·0% (98·1–99·5) 554 .. 7 .. 561 98·8% (97·4–99·5)
Roche 2 .. 974 .. 976 99·8% (99·3–100) 543 .. 18 .. 561 96·8% (95·0–98·1)
Siemens 1 .. 975 .. 976 99·9% (99·4–100) 548 .. 13 .. 561 97·7% (96·1–98·8)
Samples with complete data available taken ≥14 days post symptom onset (specificity set to ≥98%)
Abbott 19 .. 957 .. 976 98·1% (97·0–98·8) 546 .. 15 .. 561 97·3% (95·6–98·5)
DiaSorin 18 0 958 .. 976 98·2% (97·1–98·9) 544 0 17 .. 561 97·0% (95·2–98·2)
Oxford immunoassay 19 .. 957 .. 976 98·1% (97·0–98·8) 557 .. 4 .. 561 99·3% (98·2–99·8)
Roche 19 .. 957 .. 976 98·1% (97·0–98·8) 556 .. 5 .. 561 99·1% (97·9–99·7)
Siemens 15 .. 961 .. 976 98·5% (97·5–99·1) 554 .. 7 .. 561 98·8% (97·4–99·5)
Samples with complete data available taken ≥30 days post symptom onset
Abbott 1 .. 975 .. 976 99·9% (99·4–100) 458 .. 32 .. 490 93·5% (90·9–95·5)
DiaSorin 11 2 963 .. 976 98·9% (98·0–99·4) 468 6 16 .. 490 96·7% (94·7–98·1)
Oxford immunoassay 10 .. 966 .. 976 99·0% (98·1–99·5) 487 .. 3 .. 490 99·4% (98·2–99·9)
Roche 2 .. 974 .. 976 99·8% (99·3–100) 481 .. 9 .. 490 98·2% (96·5–99·2)
Siemens 1 .. 975 .. 976 99·9% (99·4–100) 482 .. 8 .. 490 98·4% (96·8–99·3)
Samples with complete data available taken ≥30 days post symptom onset (specificity set to ≥98%)
Abbott 19 .. 957 .. 976 98·1% (97·0–98·8) 482 .. 8 .. 490 98·4% (96·8–99·3)
DiaSorin 18 0 958 .. 976 98·2% (97·1–98·9) 481 0 9 .. 490 98·2% (96·5–99·2)
Oxford immunoassay 19 .. 957 .. 976 98·1% (97·0–98·8) 488 .. 2 .. 490 99·6% (98·5–100)
Roche 19 .. 957 .. 976 98·1% (97·0–98·8) 488 .. 2 .. 490 99·6% (98·5–100)
Siemens 15 .. 961 .. 976 98·5% (97·5–99·1) 485 .. 5 .. 490 99·0% (97·6–99·7)
concepts↗concept_edgesconcept_edgesSpecificity was assessed using the known negative pre-pandemic samples, and sensitivity using the known positive samples from patients who had previously had laboratory-confirmed SARS–CoV–2 infection. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).
Figure 1 Sensitivity and specificity for each assay on samples taken ≥20 days post symptom onset in patients with laboratory-confirmed SARS-CoV-2 infection for positive samples, and >6 months before the first known COVID-19 cases for negative samples
The UK MHRA performance target is shown (dark grey dashed line), including the required lower bound of the 95% CI (light grey dashed line) for sensitivity and specificity. Data are presented for 976 known-negative samples and 536 known-positive samples run on each assay. Equivocal results were excluded from the calculation of sensitivity and specificity for the DiaSorin assay (n=9). MHRA=UK Medicines and Healthcare products Regulatory Agency. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).
Figure 2 Distribution of numerical results obtained for each commercial assay
Results are represented as (A) histograms, to enable assessment of the frequency of values, and (B) dotplots, to review scatter of values, especially around thresholds. Prespecified assay thresholds are shown as dashed lines. For the purposes of plotting values on a log scale, values of zero were set to the lowest non-zero value and results of greater or less than the largest or smallest values were truncated to the largest and smallest values. Data are presented for 976 known negative samples and 536 known positive samples run across all assays. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).
concepts↗concepts↗concepts↗sensesUsing the pre-defined assay thresholds for calling test results positive or negative, only the Siemens assay and the Oxford immunoassay met the specificity and sensitivity target of at least 98%, achieving this threshold at all three timepoints assessed (≥14, ≥20, and ≥30 days post symptom onset; appendix pp 28, 29 ; figure 1 ; table ). The Roche assay also met the specificity and sensitivity target of at least 98% on samples taken at least 30 days post symptom onset ( appendix p 29 ; table ). ROC curve analysis showed that the Roche assay could also potentially meet the sensitivity and specificity targets on samples taken at least 14 and at least 20 days post symptom onset through threshold adjustment—eg, adjusting the threshold to at least 0·128 as opposed to the current threshold of more than 1·0 would result in a sensitivity and specificity of at least 98% (with a lower bound of the 95% CI of ≥96%) at 20 days or more post symptom onset ( figure 3 ; appendix p 30 for samples taken ≥14 days post symptom onset; table ). On samples taken at least 30 days post symptom onset (but not ≥14 or ≥20 days post symptom onset), the Abbott and DiaSorin assay thresholds could be optimised to achieve sensitivity and specificity of at least 98% (with a lower bound of ≥96%; appendix p 31 ; table ). Overall, on samples taken at least 30 days post symptom onset, optimisation of assay thresholds to ensure a specificity of at least 98% (with a lower bound of ≥96%) would result in sensitivities of at least 98% (with a lower bound of ≥96%) for all assays assessed ( appendix pp 31, 32 ; table ). Figure 3 ROC curves for each assay at the specification of samples taken ≥20 post symptom onset
The green shaded areas represent a target sensitivity and specificity of at least 98%. Dashed lines show the 98% sensitivity and specificity thresholds used as the standard. Assay threshold values associated with ten exemplar points on the ROC curve are shown in each panel. Data are presented for 976 known negative samples and 536 known positive samples run on each assay. ROC=receiver operating characteristic. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).
The NHSBT standards (10061-63) correlated well with the NIBSC reagents using the Oxford immunoassay ( appendix p 10 ). Replicate testing of serial dilutions of the NHSBT standards showed good reproducibility, albeit with some interesting differences observed—eg, for nucleocapsid protein assays (Abbott and Roche), readings for the NHSBT medium control (10061) dilution series were lower than those of the NHSBT low control (10063), suggesting that antibodies targeting nucleocapsid and spike proteins might be present at different titres in certain samples ( appendix p 33 ). For the Roche and Siemens assays the entire high control dilution series generated values above the assay thresholds for calling positives. For the Abbott assay, all of the medium titre dilution series generated values below the threshold for distinguishing positives.
concepts↗concepts↗concepts↗concepts↗sensesAlthough sensitivity and specificity are useful, positive and negative predictive values are more directly relevant to public health and clinical application. At 10% prevalence, the high specificity platforms (Abbott, Roche, and Siemens assays) had lower numbers of false positive tests per million tests compared with other assays (DiaSorin and Oxford immunoassay; figure 4 ; appendix p 21 ). However, the number of false negative tests per million was higher with the Abbott and DiaSorin assays and lowest with the Oxford immunoassay ( figure 4 ; appendix p 21 ). Overall, the Siemens assay had the fewest total errors per million tests for prevalences of 20% or lower ( figure 4 ; appendix p 21 ). Figure 4 False negatives, false positives, and total errors per 1 million tests
False negatives (A), false positives (B) and total errors (C) per 1 million tests, using the unadjusted thresholds (manufacturers, and Oxford immunoassay), and sensitivity and specificity for all assays for samples tested after at least 20 days post symptom onset, at population SARS-CoV-2 seroprevalence of 5%, 10%, 20%, and 50%. Abbott=SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA). DiaSorin=LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy). Oxford immunoassay=a novel 384-well format ELISA (University of Oxford, Oxford, UK). Roche=Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. SARS-CoV-2=severe acute respiratory syndrome coronavirus 2. Siemens=SARS-CoV-2 Total assay (Siemens, Munich, Germany).
concepts↗concepts↗concepts↗concepts↗concept_edgesAntibody responses increased over the first 3–4 weeks after symptom onset, with the maximum percentage of positive tests reached by day 27 across all assays ( appendix p 34 ). Antibody responses were sustained up to 73 days post symptom onset and up to 82 days post positive RT-PCR result ( appendix p 35 ); however, any declines in titres could not be specifically ruled out given that, by this stage, most quantitative results were at near-maximum values for each assay.
For 158 (29%) of 536 individuals in the positive cohort for whom disease severity data were available, we found no evidence of a difference in immunoassay sensitivity by disease severity (asymptomatic n=13 [8%], mild n=122 [77%], severe n=16 [10%], critical or death n=7 [4%]) for any of the assays assessed ( appendix p 36 ); however numbers were small in the non-mild groups.
We also undertook an analysis of discordance between assays ( appendix pp 22, 23 ). Most of the discordances were observed in samples taken before 20 days post symptom onset for SARS-CoV-2-positive patients confirmed by RT-PCR (37 [30%] of 124 samples taken <20 days post symptom onset vs 46 [9%] 538 samples taken ≥20 days post symptom onset; 27 [3%] of 976 pre-pandemic samples; p<0·0001). Three (<1%) of the samples classified as known positive (from two mild and one severe case) tested antibody-negative across all assays.
Discussion
concepts↗concepts↗concepts↗concepts↗concepts↗sensesOn a large panel of blood samples, we showed that the Siemens assay and the Oxford immunoassay both achieved sensitivity and specificity of at least 98% (with a lower bound of the 95% CI ≥96%) on samples taken at least 20 days post symptom onset without optimisation. However, all assays could potentially achieve these specifications through threshold adjustment, or by assessing samples collected at least 30 days post symptom onset, consistent with the time-dependent nature of antibody responses. Additionally, we highlighted the potentially scalable capacity of the Oxford immunoassay, which could enable centralised testing for large-scale projects (eg, national seroprevalence studies). This finding suggests that global serology testing needs can be met using different assays, mitigating against the risk of shortages, and allowing immediate deployment in laboratories with different analysers already installed for other testing purposes. limitationscontent_tagsThe optimal deployment of assays should reflect use case—eg, an optimised assay measuring antibodies to nucleocapsid protein might be most relevant for disease surveillance in the context of a widespread rollout of vaccines, which are mostly spike-protein based.
limitationsAlthough sensitivity and specificity are important, estimating the negative and positive predictive values of each assay are perhaps more useful in deciding how tests can best be used and interpreted. Most published seroprevalence studies of non-health-care workers have estimated antibody prevalences of less than 15%. 22 , 23 From our estimates, all assays would have an error rate of less than 1% at a seroprevalence of 10%, except for the DiaSorin assay. Nevertheless, this error rate would still translate into several thousand errors per million tests done for any of the assays investigated, which could have important implications at an individual level.
concepts↗concepts↗concepts↗concepts↗sensesIn our study, antibody titres within individuals were largely sustained beyond 2 months of symptom onset for all assays investigated, suggesting that serosurveillance with these assays is likely to be accurate within months of exposure at least. We did not observe any significant difference in assay sensitivity by disease severity; notably, however, the numbers in our non-mild disease category, including asymptomatic cases, were small, and we were limited by the bounds determined by the dynamic range of the assays and saturation. Inconsistencies have been observed in the association between antibody titres and disease severity in other studies, which might relate to assay target. 20 , 24 , 25 Further studies definitively assessing the longer-term trajectory of antibodies in larger longitudinal cohorts of individuals with different disease severities and taking the specific assay type into account would be of benefit.
Although all these assays can effectively detect SARS-CoV-2 antibodies, the durability and nature of immunity conferred by these antibodies remains unclear. Few data are available on immunity to seasonal human coronaviruses (alphacoronaviruses NL63 and 229E, and betacoronaviruses HKU1 and OC43). In two small, controlled, human infection models, re-challenge at 8–12 months with 229E (nine individuals) 26 and 229E-related strains (six) 27 showed protection against symptoms, but virus shedding was detected in six (67%) of nine individuals. 26 In an observational study of NL63 in Kenya, reinfections over a 6-month period occurred in 46 (28%) of 163 individuals, with reinfections within 80 days largely associated (>80% cases) with lower viral titres and fewer symptoms compared with those associated with reinfections after more than 80 days. 28 Data on Middle East respiratory syndrome (MERS) and severe acute respiratory syndrome coronavirus suggest that antibody titres wane over periods of approximately 2 years, with more severe infection resulting in more measurable and durable antibody responses in MERS infection. 29 , 30 Animal data (re-challenge in four rhesus macaques) suggest protection from re-infection with SARS-CoV-2 extends to at least 28 days post infection. 31 The relationship between SARS-CoV-2 antibodies and neutralising activity remains an important research question. Because the spike protein is responsible for enabling cell entry, the spike protein appears to be the main antigen responsible for eliciting neutralising antibodies. In SARS-CoV-2, a correlation between anti-spike antibody titres when trimeric versions of spike are used as the antigen and neutralising capacity appears to be emerging. 24 , 32 If SARS-CoV-2 antibody titres are shown to clearly correlate directly or indirectly with immunity to infection or attenuation of disease severity, then more accurately evaluating titres by optimising the dynamic range of assays might become relevant.
phenomenonmechanismscontrolsconcepts↗concepts↗sensessensessensesconcept_edgescontent_tagscontent_tags↗content_tags↗Notably, no clear gold standard exists against which to assess these antibody tests. RT-PCR positivity is a proxy for the expected presence of antibody, but negative antibody tests in RT-PCR-positive individuals could either reflect antibody test performance, or alternatively be explained by a failure to mount a measurable systemic IgM or IgG antibody response (eg, in cases of immunocompromised individuals), effective infection clearance through other immune mechanisms (eg, T cells), an issue with the sample (eg, a major interferent), or through a false-positive RT-PCR test in individuals who have not had SARS-CoV-2 infection. limitationsThe three samples that were negative by all assays in our study could plausibly have had a biological absence of antibodies.
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗Our study was limited by sample volumes, especially given the constraints imposed by dead volume requirements for liquid handling, and we were unable to do repeat analyses. Subgroup analysis by timing of collection or disease severity was constrained by small numbers and larger studies would be of value. Neutralisation and pseudo-neutralisation assays provide an in-vitro approach to investigating antibodies to help address the question as to whether antibody titres correlate with functionality; this work is ongoing. Longitudinal prospective clinical studies are also needed to investigate whether repeat infections can arise in antibody-positive individuals. limitationscontent_tagsOur sample sets might under-represent some ethnic groups, did not include children, and captured insufficient clinical meta-data to characterise features (eg, being immunocompromised), which might explain the presence of false-negative antibody results in individuals with RT-PCR-confirmed infection. Additional performance assessments in these groups would be warranted. Although assessments of precision and reproducibility have to some extent been carried out by the immunoassay manufacturers, a complete validation of the individual assays was beyond the scope of our study. We did not investigate our large negative sample set for the presence of antibodies against seasonal coronaviruses, the presence of which might have accounted for some of the false-positive results. However, previous manufacturers' assessments for the Abbott and Roche immunoassays, and our assessment as part of this study for the Oxford immunoassay on small panels of sera taken from patients with respiratory virus infections including seasonal coronavirus infections, suggest good analytical specificity.
mechanismscontrolsconcepts↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationssensescontent_tagsIn conclusion, we characterised the performance of four commercial antibody platforms and a new ELISA from an academic research facility, all of which achieved the desired specificity, and with opportunities to optimise sensitivity for the Abbott, DiaSorin, and Roche assays through changes to thresholds and sample timing, although any post-hoc changes to thresholds would require repeat verification of sensitivity and specificity. Assay selection should carefully reflect use case. This study represents a benchmark for future assessments of serological platforms. Such assays will be an important part of the clinical and research landscape in guiding public health policy, with effects to be delivered at the individual level and population level.
limitationscontent_tagsThis online publication has been corrected. The corrected version first appeared at thelancet.com/infection on November 25, 2020
Data sharing
Available metadata for each sample in the study, including raw immunoassay results and interpretation, are presented as online datasets .
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 10 of 12 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | FIVE IMMUNOASSAYS MEASURING ONE CONSTRUCT ON ONE SAMPLE SET - 976 pre-pandemic and 536 PCR-confirmed bloods run through four commercial platforms and a novel ELISA, with the thresholds treated as adjustable | object_of_study | the paper's words | |
| research_question | how five widely deployed SARS-CoV-2 antibody assays compare when run on identical samples, and what thresholds each needs to reach 98% sensitivity and specificity | research_question | the paper's words | |
| contribution | the first head-to-head assessment of four commercial platforms plus a scalable 384-well ELISA on one sample set, with ROC-derived thresholds and an error-rate projection per million tests | contribution | the paper's words | |
| claim_type | method | claim_type | the paper's words | |
| phenomenon | an instrument's measured accuracy as a property of the cut point somebody chose, in the absence of anything to check the instrument against | phenomenon | the paper's words | |
| method | comparative also in: comparative | method | the paper's words | |
| mechanisms | THE PAPER SAYS OUTRIGHT THAT THERE IS NOTHING TO MEASURE THE INSTRUMENTS AGAINST: `Notably, no clear gold standard exists against which to assess these antibody tests. RT-PCR positivity is a proxy for the expected presence of antibody, but negative antibody tests in RT-PCR-positive individuals could either reflect antibody test performance, or alternatively be explained by a failure to mount a measurable systemic IgM or IgG antibody response ... effective infection clearance through other immune mechanisms (eg, T cells), an issue with the sample ... or through a false-positive RT-PCR test`. FIVE WAYS TO GET A DISCORDANT RESULT AND ONLY ONE OF THEM IS THE ASSAY BEING WRONG | mechanisms[0] | 0.95 | the tagger's words |
| mechanisms | AND THE THRESHOLD IS WHAT MOVES, NOT THE ASSAY: at the manufacturers' cut points the five assays span 92.7% to 99.1% sensitivity, and then `All assays achieved a sensitivity of at least 98% with thresholds optimised to achieve a specificity of at least 98% on samples taken 30 days or more post symptom onset`. Same five machines, same 1,512 samples, and every one of them passes once someone else chooses the cut point | mechanisms[1] | 0.95 | the tagger's words |
| mechanisms | and the authors add the caveat that makes it honest rather than circular: `opportunities to optimise sensitivity for the Abbott, DiaSorin, and Roche assays through changes to thresholds and sample timing, although any post-hoc changes to thresholds would require repeat verification of sensitivity and specificity` | mechanisms[2] | 0.95 | the tagger's words |
| mechanisms | AND THE COST OF THE CHOICE IS COUNTED IN PEOPLE: `at a 10% seroprevalence, the Siemens assay would generate an estimated 2800 total errors per million tests, versus the DiaSorin assay at a rate of 13 700 total errors per million tests`. A five-fold difference in wrong answers, from a choice of platform | mechanisms[3] | 0.95 | the tagger's words |
| mechanisms | AND THE PRIOR LITERATURE HAS THE RIFLE PROBLEM: `For the Abbott assay, sensitivity RANGED FROM 93.4% TO 100% ... (70 to 680 samples) ... for the DiaSorin assay, we found A single sensitivity estimate of 94.4% ... (18 SAMPLES)`, and `No published studies assessed all four commercial platforms using the same sample sets`. The spread was never a property of the assays | mechanisms[4] | 0.95 | the tagger's words |
| limitations | ONE INSTRUMENT WAS DROPPED FROM THE COMPARISON BEFORE IT BEGAN, ON THE STRENGTH OF A PRIOR REPORT: `the sensitivity of the EUROIMMUN assay has been reported as being approximately 90%. We therefore replaced this with the SARS-CoV-2 Total assay (Siemens ...)`. A selection decision that shapes what the benchmark can find, made in the open and worth reading next to the finding that prior estimates ranged from 93.4% to 100% on samples as small as eighteen | limitations[0] | 0.95 | the tagger's words |
| limitations | AND THE ASSESSMENT CRITERIA THEMSELVES HAD BEEN RELAXED: `governments, regulators, and clinical laboratories have had to balance the urgent need ... with the few data available on assay performance, leading to a relaxation of typical assessment criteria (eg, the US Food and Drug Administration's [FDA] Emergency Use Authorization programme)` | limitations[1] | 0.95 | the tagger's words |
| limitations | AND THE ANSWER DEPENDS ON WHAT THE TEST IS FOR: `The optimal deployment of assays should reflect use case - eg, an optimised assay measuring antibodies to NUCLEOCAPSID protein might be most relevant for disease surveillance in the context of A widespread rollout of vaccines, which are mostly spike-protein based`. The best instrument changes when the population changes | limitations[2] | 0.95 | the tagger's words |
| limitations | and the headline metrics are not the useful ones: `Although sensitivity and specificity are important, estimating the negative and positive predictive values of each assay are perhaps more useful in deciding how tests can best be used and interpreted` - because those depend on prevalence, which is what the assays are being used to estimate | limitations[3] | 0.95 | the tagger's words |
| limitations | WHO IS NOT IN THE SAMPLE, STATED: `Our sample sets might under-represent some ethnic groups, did not include children, and captured insufficient clinical meta-data to characterise features (eg, being immunocompromised), which might explain the presence of false-negative antibody results in individuals with RT-PCR-confirmed infection` | limitations[4] | 0.95 | the tagger's words |
| limitations | and three discordant samples are left explicitly undecided rather than counted as failures: `The three samples that were negative by all assays in our study could plausibly have had a biological absence of antibodies` | limitations[5] | 0.95 | the tagger's words |
| limitations | THE PUBLISHED RECORD OF THIS PAPER CARRIES ITS OWN CORRECTION: `This online publication has been corrected. The corrected version first appeared at thelancet.com/infection on November 25, 2020`, and the PMC record we staged is VERSION 2. The project's distinction applies - a correction re-grounds a record, it does not retire it | limitations[6] | 0.95 | the tagger's words |
| controls | 976 pre-pandemic blood samples collected between September 2014 and October 2016 as the negative panel eliminates: specificity estimated on samples that might contain SARS-CoV-2 antibodies - the samples predate the virus · outcome: held | controls[0] | 0.95 | the tagger's words |
| controls | running all five assays on one identical sample set eliminates: the between-study variation that made the prior literature unusable: 'No published studies assessed all four commercial platforms using the same sample sets' · outcome: held | controls[1] | 0.95 | the tagger's words |
| controls | a reference standard to score the assays against eliminates: nothing - it does not exist: 'no clear gold standard exists against which to assess these antibody tests', and RT-PCR positivity is only a proxy with four named failure modes · outcome: absent-not-in-staged-text | controls[2] | 0.95 | the tagger's words |
| controls | testing for antibodies to seasonal coronaviruses in the negative panel eliminates: nothing here - declared not done: the authors say they 'did not investigate our large negative sample set for the presence of antibodies against seasonal coronaviruses, the presence of which might have accounted for some of the false-positive results' · outcome: absent-declared-unreported | controls[3] | 0.95 | the tagger's words |
| concepts | gold standard also in: gold standard | concepts[0] | 0.95 | the paper's words |
| concepts | RT-PCR positivity also in: RT-PCR positivity | concepts[1] | 0.95 | the paper's words |
| concepts | Siemens assay also in: Siemens assay | concepts[2] | 0.95 | the paper's words |
| concepts | DiaSorin assay also in: DiaSorin assay | concepts[3] | 0.95 | the paper's words |
| concepts | EUROIMMUN assay also in: EUROIMMUN assay | concepts[4] | 0.95 | the paper's words |
| concepts | global pandemic also in: global pandemic | concepts[5] | 0.95 | the paper's words |
| concepts | Serology also in: Serology | concepts[6] | 0.95 | the paper's words |
| concepts | population-level surveillance also in: population-level surveillance | concepts[7] | 0.95 | the paper's words |
| concepts | individual-level risk assessment also in: individual-level risk assessment | concepts[8] | 0.95 | the paper's words |
| concepts | head-to-head assessment also in: head-to-head assessment | concepts[9] | 0.95 | the paper's words |
| concepts | four high-throughput commercial SARS-CoV-2 antibody immunoassays | concepts[10] | 0.95 | the paper's words |
| concepts | novel 384-well ELISA also in: novel 384-well ELISA | concepts[11] | 0.95 | the paper's words |
| concepts | Oxford immunoassay also in: Oxford immunoassay | concepts[12] | 0.95 | the paper's words |
| concepts | pre-pandemic blood samples also in: pre-pandemic blood samples | concepts[13] | 0.95 | the paper's words |
| concepts | laboratory-confirmed SARS-CoV-2 infection | concepts[14] | 0.95 | the paper's words |
| concepts | Receiver operating characteristic (ROC) curves | concepts[15] | 0.95 | the paper's words |
| concepts | assay thresholds also in: assay thresholds | concepts[16] | 0.95 | the paper's words |
| concepts | sensitivity also in: sensitivity | concepts[17] | 0.95 | the paper's words |
| concepts | specificity also in: specificity | concepts[18] | 0.95 | the paper's words |
| concepts | Severe acute respiratory syndrome coronavirus 2 | concepts[19] | 0.95 | the paper's words |
| concepts | SARS-CoV-2 also in: SARS-CoV-2 | concepts[20] | 0.95 | the paper's words |
| concepts | direct detection of the virus also in: direct detection of the virus | concepts[21] | 0.95 | the paper's words |
| concepts | real-time RT-PCR also in: real-time RT-PCR | concepts[22] | 0.95 | the paper's words |
| concepts | presence of antibodies also in: presence of antibodies | concepts[23] | 0.95 | the paper's words |
| concepts | specific types of antibody also in: specific types of antibody | concepts[24] | 0.95 | the paper's words |
| concepts | total antibody also in: total antibody | concepts[25] | 0.95 | the paper's words |
| concepts | early acute infection also in: early acute infection | concepts[26] | 0.95 | the paper's words |
| concepts | population exposure also in: population exposure | concepts[27] | 0.95 | the paper's words |
| concepts | global laboratory infrastructures also in: global laboratory infrastructures | concepts[28] | 0.95 | the paper's words |
| concepts | population-wide testing also in: population-wide testing | concepts[29] | 0.95 | the paper's words |
| concepts | typical assessment criteria also in: typical assessment criteria | concepts[30] | 0.95 | the paper's words |
| concepts | Emergency Use Authorization programme | concepts[31] | 0.95 | the paper's words |
| concepts | regulatory approvals for emergency use | concepts[32] | 0.95 | the paper's words |
| concepts | PubMed, BioRxiv, and MedRxiv also in: PubMed, BioRxiv, and MedRxiv | concepts[33] | 0.95 | the paper's words |
| concepts | full-text reviews also in: full-text reviews | concepts[34] | 0.95 | the paper's words |
| concepts | same sample sets also in: same sample sets | concepts[35] | 0.95 | the paper's words |
| concepts | SARS-CoV-2 Total assay also in: SARS-CoV-2 Total assay | concepts[36] | 0.95 | the paper's words |
| concepts | large, single sample set of more than 1500 samples | concepts[37] | 0.95 | the paper's words |
| concepts | global installed base also in: global installed base | concepts[38] | 0.95 | the paper's words |
| concepts | assay threshold adjustment also in: assay threshold adjustment | concepts[39] | 0.95 | the paper's words |
| concepts | symptom-to-sample timings also in: symptom-to-sample timings | concepts[40] | 0.95 | the paper's words |
| concepts | serological testing resource also in: serological testing resource | concepts[41] | 0.95 | the paper's words |
| concepts | blood (plasma and serum) sample collection | concepts[42] | 0.95 | the paper's words |
| concepts | pre-pandemic samples from healthy individuals aged at least 18 years | concepts[43] | 0.95 | the paper's words |
| concepts | samples from individuals aged at least 18 years with laboratory-confirmed SARS-CoV-2 infection | concepts[44] | 0.95 | the paper's words |
| concepts | known negative samples also in: known negative samples | concepts[45] | 0.95 | the paper's words |
| concepts | known positive samples also in: known positive samples | concepts[46] | 0.95 | the paper's words |
| concepts | de-duplicated by individual also in: de-duplicated by individual | concepts[47] | 0.95 | the paper's words |
| concepts | latest sample from each individual also in: latest sample from each individual | concepts[48] | 0.95 | the paper's words |
| concepts | eight-point 1:2 dilution series also in: eight-point 1:2 dilution series | concepts[49] | 0.95 | the paper's words |
| concepts | SARS-CoV-2 IgG antibodies also in: SARS-CoV-2 IgG antibodies | concepts[50] | 0.95 | the paper's words |
| concepts | optical density also in: optical density | concepts[51] | 0.95 | the paper's words |
| concepts | protocol before implementing the study | concepts[52] | 0.95 | the paper's words |
| concepts | nucleocapsid protein also in: nucleocapsid protein | concepts[53] | 0.95 | the paper's words |
| concepts | spike protein-based targets also in: spike protein-based targets | concepts[54] | 0.95 | the paper's words |
| concepts | IgG assays also in: IgG assays | concepts[55] | 0.95 | the paper's words |
| concepts | total antibody assays also in: total antibody assays | concepts[56] | 0.95 | the paper's words |
| concepts | trained laboratory staff also in: trained laboratory staff | concepts[57] | 0.95 | the paper's words |
| concepts | specified controls and calibrants also in: specified controls and calibrants | concepts[58] | 0.95 | the paper's words |
| concepts | unique study barcodes also in: unique study barcodes | concepts[59] | 0.95 | the paper's words |
| concepts | sample metadata also in: sample metadata | concepts[60] | 0.95 | the paper's words |
| concepts | indirect ELISA also in: indirect ELISA | concepts[61] | 0.95 | the paper's words |
| concepts | serum IgG also in: serum IgG | concepts[62] | 0.95 | the paper's words |
| concepts | trimeric spike protein also in: trimeric spike protein | concepts[63] | 0.95 | the paper's words |
| concepts | Readouts also in: Readouts | concepts[64] | 0.95 | the paper's words |
| concepts | fluorescent signal also in: fluorescent signal | concepts[65] | 0.95 | the paper's words |
| concepts | natural cubic spline-based regression model | concepts[66] | 0.95 | the paper's words |
| concepts | across-plate and across-batch comparisons | concepts[67] | 0.95 | the paper's words |
| concepts | 384-well plates also in: 384-well plates | concepts[68] | 0.95 | the paper's words |
| concepts | Janus automated liquid handler also in: Janus automated liquid handler | concepts[69] | 0.95 | the paper's words |
| concepts | EnVision 2104 multilabel plate reader | concepts[70] | 0.95 | the paper's words |
| concepts | low input sample volumes also in: low input sample volumes | concepts[71] | 0.95 | the paper's words |
| concepts | dead volume requirements also in: dead volume requirements | concepts[72] | 0.95 | the paper's words |
| concepts | sample size also in: sample size | concepts[73] | 0.95 | the paper's words |
| concepts | 80% power also in: 80% power | concepts[74] | 0.95 | the paper's words |
| concepts | exact binomial 95% CIs also in: exact binomial 95% CIs | concepts[75] | 0.95 | the paper's words |
| concepts | UK MHRA guidance also in: UK MHRA guidance | concepts[76] | 0.95 | the paper's words |
| concepts | primary analysis also in: primary analysis | concepts[77] | 0.95 | the paper's words |
| concepts | secondary analyses also in: secondary analyses | concepts[78] | 0.95 | the paper's words |
| concepts | receiver operator characteristic (ROC) curves | concepts[79] | 0.95 | the paper's words |
| concepts | symptom severity also in: symptom severity | concepts[80] | 0.95 | the paper's words |
| concepts | asymptomatic, mild, severe, critical or death | concepts[81] | 0.95 | the paper's words |
| concepts | Positive and negative predictive values | concepts[82] | 0.95 | the paper's words |
| concepts | population prevalences of 5%, 10%, 20%, and 50% | concepts[83] | 0.95 | the paper's words |
| concepts | antibody titres also in: antibody titres | concepts[84] | 0.95 | the paper's words |
| concepts | spline curves also in: spline curves | concepts[85] | 0.95 | the paper's words |
| concepts | Akaike Information Criterion also in: Akaike Information Criterion | concepts[86] | 0.95 | the paper's words |
| concepts | external review group also in: external review group | concepts[87] | 0.95 | the paper's words |
| concepts | known negative pre-pandemic samples also in: known negative pre-pandemic samples | concepts[88] | 0.95 | the paper's words |
| concepts | known positive samples from patients who had previously had laboratory-confirmed SARS–CoV–2 infection | concepts[89] | 0.95 | the paper's words |
| concepts | liquid handling failures also in: liquid handling failures | concepts[90] | 0.95 | the paper's words |
| concepts | pre-defined assay thresholds also in: pre-defined assay thresholds | concepts[91] | 0.95 | the paper's words |
| concepts | threshold adjustment also in: threshold adjustment | concepts[92] | 0.95 | the paper's words |
| concepts | samples taken at least 30 days post symptom onset | concepts[93] | 0.95 | the paper's words |
| concepts | false positive tests per million tests | concepts[94] | 0.95 | the paper's words |
| concepts | false negative tests per million also in: false negative tests per million | concepts[95] | 0.95 | the paper's words |
| concepts | total errors per million tests also in: total errors per million tests | concepts[96] | 0.95 | the paper's words |
| concepts | prevalences of 20% or lower also in: prevalences of 20% or lower | concepts[97] | 0.95 | the paper's words |
| concepts | Antibody responses also in: Antibody responses | concepts[98] | 0.95 | the paper's words |
| concepts | first 3–4 weeks after symptom onset also in: first 3–4 weeks after symptom onset | concepts[99] | 0.95 | the paper's words |
| concepts | day 27 also in: day 27 | concepts[100] | 0.95 | the paper's words |
| concepts | discordance between assays also in: discordance between assays | concepts[101] | 0.95 | the paper's words |
| concepts | centralised testing also in: centralised testing | concepts[102] | 0.95 | the paper's words |
| concepts | use case also in: use case | concepts[103] | 0.95 | the paper's words |
| concepts | disease surveillance also in: disease surveillance | concepts[104] | 0.95 | the paper's words |
| concepts | widespread rollout of vaccines also in: widespread rollout of vaccines | concepts[105] | 0.95 | the paper's words |
| concepts | spike-protein based also in: spike-protein based | concepts[106] | 0.95 | the paper's words |
| concepts | dynamic range of the assays also in: dynamic range of the assays | concepts[107] | 0.95 | the paper's words |
| concepts | neutralising activity also in: neutralising activity | concepts[108] | 0.95 | the paper's words |
| concepts | neutralising antibodies also in: neutralising antibodies | concepts[109] | 0.95 | the paper's words |
| concepts | immunity to infection also in: immunity to infection | concepts[110] | 0.95 | the paper's words |
| concepts | sample volumes also in: sample volumes | concepts[111] | 0.95 | the paper's words |
| concepts | dead volume requirements for liquid handling | concepts[112] | 0.95 | the paper's words |
| concepts | Neutralisation and pseudo-neutralisation assays | concepts[113] | 0.95 | the paper's words |
| concepts | Longitudinal prospective clinical studies | concepts[114] | 0.95 | the paper's words |
| concepts | insufficient clinical meta-data also in: insufficient clinical meta-data | concepts[115] | 0.95 | the paper's words |
| concepts | analytical specificity also in: analytical specificity | concepts[116] | 0.95 | the paper's words |
| concepts | post-hoc changes to thresholds also in: post-hoc changes to thresholds | concepts[117] | 0.95 | the paper's words |
| concepts | repeat verification of sensitivity and specificity | concepts[118] | 0.95 | the paper's words |
| concepts | benchmark for future assessments of serological platforms | concepts[119] | 0.95 | the paper's words |
| concepts | Assay selection also in: Assay selection | concepts[120] | 0.95 | the paper's words |
| unhoused_relations | direct detection of the virus → real-time RT-PCR Direct viral detection is performed with real-time RT-PCR; performed-with is absent from the eight governed types. — the relation stated: Introduction Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has emerged as a novel human pathogen, causing a global pandemic in 2020, with more than 25 million confirmed infections and more than 840 000 deaths to date. 1 Testing and case ascertainment have been crucial to controlling virus transmission and in developing public health and political strategies to mitigate the effect of this pathogen. SARS-CoV-2 testing broadly takes two forms: first, direct detection of the virus in respiratory samples with real-time RT-PCR; and second, by using serology to investigate the presence of antibodies. 2 Immunoassays detect either specific types of antibody (eg, IgM or IgG) or total antibody. SARS-CoV-2 antibodies typically start to appear at least 5–7 days post infection 3 and are therefore an unreliable marker for early acute infection. The degree and duration of immunity that antibodies confer are unclear. A prominent use for serological testing has therefore been at a population level, for informing the extent of population exposure. Other uses include assessing risk of infection at an individual level and to support research and development (eg, quantifying antibody responses in vaccine trials). 4 Several manufacturers have developed immunoassays compatible with global laboratory infrastructures, enabling widespread testing of hundreds to thousands of samples per day. However, the scale-up required for regular population-wide testing (eg, every few weeks or months) might exceed the capacity of commercial platforms. To date, few thorough, direct assessments of immunoassay performance on large sample sets have been done, and governments, regulators, and clinical laboratories have had to balance the urgent need to facilitate the demand for serological testing with the few data available on assay performance, leading to a relaxation of typical assessment criteria (eg, the US Food and Drug Administration's [FDA] Emergency Use Authorization programme) | unhoused_relations[0] | 0.95 | the tagger's words |
| unhoused_relations | serology → presence of antibodies Serology is used to investigate antibody presence; investigates is absent from the eight types. — the relation stated: Introduction Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has emerged as a novel human pathogen, causing a global pandemic in 2020, with more than 25 million confirmed infections and more than 840 000 deaths to date. 1 Testing and case ascertainment have been crucial to controlling virus transmission and in developing public health and political strategies to mitigate the effect of this pathogen. SARS-CoV-2 testing broadly takes two forms: first, direct detection of the virus in respiratory samples with real-time RT-PCR; and second, by using serology to investigate the presence of antibodies. 2 Immunoassays detect either specific types of antibody (eg, IgM or IgG) or total antibody. SARS-CoV-2 antibodies typically start to appear at least 5–7 days post infection 3 and are therefore an unreliable marker for early acute infection. The degree and duration of immunity that antibodies confer are unclear. A prominent use for serological testing has therefore been at a population level, for informing the extent of population exposure. Other uses include assessing risk of infection at an individual level and to support research and development (eg, quantifying antibody responses in vaccine trials). 4 Several manufacturers have developed immunoassays compatible with global laboratory infrastructures, enabling widespread testing of hundreds to thousands of samples per day. However, the scale-up required for regular population-wide testing (eg, every few weeks or months) might exceed the capacity of commercial platforms. To date, few thorough, direct assessments of immunoassay performance on large sample sets have been done, and governments, regulators, and clinical laboratories have had to balance the urgent need to facilitate the demand for serological testing with the few data available on assay performance, leading to a relaxation of typical assessment criteria (eg, the US Food and Drug Administration's [FDA] Emergency Use Authorization programme) | unhoused_relations[1] | 0.95 | the tagger's words |
| unhoused_relations | nucleocapsid protein → IgG assays The assay taxonomy crosses antigen target and antibody-isotype class; targets is absent from the eight governed types. — the relation stated: Procedures The installed base of analysers running the investigated assays is global, making their widespread deployment feasible. Two assays targeted the nucleocapsid protein (Abbott and Roche) and two had spike protein-based targets (DiaSorin and Siemens); two were IgG assays (Abbott and DiaSorin), and two were total antibody assays (Roche and Siemens). Manufacturers' sensitivity estimates ranged from 96·8% to 100% on samples taken at least 14 days post symptom onset or RT-PCR, and specificity estimates ranged from 98·5% to 99·8% on pre-pandemic samples ( appendix p 14 ). 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 Assays were done in accordance with the manufacturers' instructions by trained laboratory staff in laboratories that were accredited by the UK Accreditation Service, on appropriate analysers, and with the specified controls and calibrants, using thresholds for calling positives and negatives set by the manufacturer for testing in the UK ( appendix p 14 ), but also reporting quantitative values. Samples were assigned unique study barcodes for this study, and laboratory staff doing the assays did not have access to sample metadata before running the assays | unhoused_relations[2] | 0.95 | the tagger's words |
| unhoused_relations | trimeric spike protein → Oxford immunoassay Trimeric spike is the antigen measured by the Oxford immunoassay; target-of is absent from the eight types. | unhoused_relations[3] | 0.95 | the tagger's words |
| unhoused_relations | EUROIMMUN assay → SARS-CoV-2 Total assay EUROIMMUN was replaced with Siemens in the head-to-head comparison; replaces or selected-instead-of is absent from the eight types. — the relation stated: Research in context Evidence before this study Substantial global interest exists in the use of serology to enable population-level surveillance of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection and to inform individual-level management and risk stratification; however, it is unclear which widely available SARS-CoV-2 serological immunoassays perform to the standards required to meet these needs. The most widely used commercial assays that have obtained regulatory approvals for emergency use in the USA and the EU include the SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA), LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy), Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland), and the anti-SARS-COV-2 ELISA (IgG; EUROIMMUN, Lübeck, Germany). However, the sensitivity of the EUROIMMUN assay has been reported as being approximately 90%. We therefore replaced this with the SARS-CoV-2 Total assay (Siemens, Munich, Germany) in a head-to-head comparison. In addition to reviewing performance assessments undertaken by each manufacturer, we searched PubMed, BioRxiv, and MedRxiv up to May 31, 2020, using the following search terms: (SARS-CoV-2) AND ([ELISA] OR [EIA] OR [CLIA] OR [FIA] OR [IFA] OR [IgG]). Details and expanded PubMed search terms are given in the appendix (pp 7, 8) . We also reviewed any investigations of relevant assays by Public Health England (PHE) up to May 31, 2020. Of 423 studies assessed, full-text reviews were done for 124 articles. Eight studies provided data on sensitivity and specificity for the Abbott (four studies) or DiaSorin (five) assays; PHE investigations were undertaken for the Abbott and Roche assays. For the Abbott assay, sensitivity ranged from 93·4% to 100% for samples taken at least 14 days post symptom onset (70 to 680 samples) and specificity from 95·1% to 100% (nine to 1020); for the DiaSorin assay, we found a single sensitivity estimate of 94·4% on samples taken at least 14 days post symptom onset (18 samples) and specificity ranged from 94·9% to 100% (69 to 1140 samples). For the Roche assay, we found a single sensitivity estimate of 87·0% on samples taken at least 14 days post symptom onset (n=77) and a single specificity estimate of 100% (n=472). No published studies assessed all four commercial platforms using the same sample sets | unhoused_relations[4] | 0.95 | the tagger's words |
| unhoused_relations | post-hoc changes to thresholds → repeat verification of sensitivity and specificity Any post-hoc threshold change requires repeat verification; prerequisite-for is absent from the eight governed types. | unhoused_relations[5] | 0.95 | the tagger's words |
| unhoused_relations | use case → Assay selection Assay selection should reflect the use case; should-be-governed-by is absent from the eight types. | unhoused_relations[6] | 0.95 | the tagger's words |
| senses | `gold standard` (absent) - `no clear gold standard exists against which to assess these antibody tests`, and the substitute is explicitly a proxy: `RT-PCR positivity is a proxy for the expected presence of antibody` | senses[0] | 0.95 | the tagger's words |
| senses | `sensitivity` (manufacturer sense) vs `sensitivity` (ROC-optimised sense) - the same word for two numbers on the same machine and the same samples. Abbott is 92.7% under one and at least 98% under the other | senses[1] | 0.95 | the tagger's words |
| senses | `false negative` - here NOT a property of the assay: an antibody-negative PCR-positive result can be immunocompromise, T-cell clearance, a bad sample, or a FALSE-POSITIVE PCR. Four alternatives to the instrument being wrong | senses[2] | 0.95 | the tagger's words |
| senses | Head-to-head benchmark - all platforms are scored on one large, identical sample set rather than compared across separate studies | senses[3] | 0.95 | the tagger's words |
| senses | Sensitivity reference - antibody detection among RT-PCR-confirmed post-symptom samples, with RT-PCR acknowledged as only a proxy | senses[4] | 0.95 | the tagger's words |
| senses | Specificity reference - absence of positive calls among samples collected years before SARS-CoV-2 appeared | senses[5] | 0.95 | the tagger's words |
| senses | Threshold-dependent performance - manufacturer cut points and ROC-optimised cut points produce different sensitivity-specificity pairs | senses[6] | 0.95 | the tagger's words |
| senses | Predictive-value framing - expected false positives, false negatives, and total errors depend on population prevalence | senses[7] | 0.95 | the tagger's words |
| senses | Use-case-dependent assay choice - nucleocapsid assays become useful for surveillance when vaccination is spike-protein based | senses[8] | 0.95 | the tagger's words |
| senses | Antibody-versus-immunity boundary - assays detect antibodies, while durability and protective meaning remain uncertain | senses[9] | 0.95 | the tagger's words |
| senses | Post-hoc threshold optimisation - any changed cut point requires renewed verification of both sensitivity and specificity | senses[10] | 0.95 | the tagger's words |
| alternatives | claim_type: method a second reading reaching the same value. Recorded: method — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., | alternatives.claim_type | 0.8 | the tagger's words |
| alternatives | method: diagnostic-accuracy-benchmark a second reading DIFFERING from the recorded value. Recorded: comparative — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., | alternatives.method | 0.85 | the tagger's words |
| alternatives | phenomenon: the sensitivity and specificity of five SARS-CoV-2 antibody immunoassays measured on identical sample sets a second reading DIFFERING from the recorded value. Recorded: an instrument's measured accuracy as a property of the cut point somebody chose, in the absence of anything to check the — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., | alternatives.phenomenon | 0.9 | the tagger's words |
| claim_relations | extends → the eight prior studies of the Abbott and DiaSorin assays, whose sensitivity estimates ranged from 93.4% to 100% on sample sets of eighteen to 680 | claim_relations[0] | 0.95 | the tagger's words |
| claim_relations | contrasts-with → the manufacturers' own threshold settings, under which only two of the five assays reach 98% sensitivity and specificity | claim_relations[1] | 0.95 | the tagger's words |
| concept_edges | Siemens assay —distinct-from→ DiaSorin assay strength of the relation: 0.95 · note: 'at a 10% seroprevalence, the Siemens assay would generate an estimated 2800 total errors per million tests, versus the DiaSorin assay at a rate of 13 700 total errors per million tests'. Two instruments for one construct, run on the same 1,512 samples, differing by a factor of five in the number of people they get wrong
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span compares the two assays' error rates; distinct-from is loose. also in: Siemens assay · DiaSorin assay | concept_edges[0] | 0.95 | the tagger's words |
| concept_edges | gold standard —operationalised-as→ RT-PCR positivity strength of the relation: 0.9 · note: THE CUE AND THE TYPE DISAGREE ON PURPOSE. The cue is KINDHOOD 'is a', in 'RT-PCR positivity is a proxy for the expected presence of antibody' - but the relation is not kindhood, it is substitution: the reference standard is absent ('no clear gold standard exists') and RT-PCR positivity stands in for it. Weight 0.9 because the paper states four ways the proxy and the target can disagree without either being wrong
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but 'no clear gold standard exists' and RT-PCR positivity is 'a proxy'; operationalised-as is loose. also in: gold standard · RT-PCR positivity | concept_edges[1] | 0.95 | the tagger's words |
| concept_edges | SARS-CoV-2 —same-as→ Severe acute respiratory syndrome coronavirus 2 the relation itself is separately evidenced · strength of the relation: 1 · note: Parenthetical abbreviation of the expanded pathogen name. | concept_edges[2] | 0.95 | the tagger's words |
| concept_edges | novel 384-well ELISA —same-as→ Oxford immunoassay the relation itself is separately evidenced · strength of the relation: 1 · note: The paper introduces the ELISA with Oxford immunoassay as its parenthetical name. · the relation stated: Summary Background Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has caused a global pandemic in 2020. Testing is crucial for mitigating public health and economic effects. Serology is considered key to population-level surveillance and potentially individual-level risk assessment. However, immunoassay performance has not been compared on large, identical sample sets. We aimed to investigate the performance of four high-throughput commercial SARS-CoV-2 antibody immunoassays and a novel 384-well ELISA. Methods We did a head-to-head assessment of SARS-CoV-2 IgG assay (Abbott, Chicago, IL, USA), LIAISON SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Saluggia, Italy), Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland), SARS-CoV-2 Total assay (Siemens, Munich, Germany), and a novel 384-well ELISA (the Oxford immunoassay). We derived sensitivity and specificity from 976 pre-pandemic blood samples (collected between Sept 4, 2014, and Oct 4, 2016) and 536 blood samples from patients with laboratory-confirmed SARS-CoV-2 infection, collected at least 20 days post symptom onset (collected between Feb 1, 2020, and May 31, 2020). Receiver operating characteristic (ROC) curves were used to assess assay thresholds. Findings At the manufacturers' thresholds, for the Abbott assay sensitivity was 92·7% (95% CI 90·2–94·8) and specificity was 99·9% (99·4–100%); for the DiaSorin assay sensitivity was 96·2% (94·2–97·7) and specificity was 98·9% (98·0–99·4); for the Oxford immunoassay sensitivity was 99·1% (97·8–99·7) and specificity was 99·0% (98·1–99·5); for the Roche assay sensitivity was 97·2% (95·4–98·4) and specificity was 99·8% (99·3–100); and for the Siemens assay sensitivity was 98·1% (96·6–99·1) and specificity was 99·9% (99·4–100%). All assays achieved a sensitivity of at least 98% with thresholds optimised to achieve a specificity of at least 98% on samples taken 30 days or more post symptom onset. Interpretation Four commercial, widely available assays and a scalable 384-well ELISA can be used for SARS-CoV-2 serological testing to achieve sensitivity and specificity of at least 98%. The Siemens assay and Oxford immunoassay achieved these metrics without further optimisation. This benchmark study in immunoassay assessment should enable refinements of testing strategies and the best use of serological testing resource to benefit individuals and population health. Funding Public Health England and UK National Institute for Health Research also in: novel 384-well ELISA · Oxford immunoassay | concept_edges[3] | 0.95 | the tagger's words |
| concept_edges | Oxford immunoassay —is-a→ indirect ELISA the relation itself is separately evidenced · strength of the relation: 1 · note: Direct class statement. also in: Oxford immunoassay · indirect ELISA | concept_edges[4] | 0.95 | the tagger's words |
| concept_edges | pre-pandemic samples from healthy individuals aged at least 18 years —part-of→ blood (plasma and serum) sample collection the relation itself is separately evidenced · strength of the relation: 1 · note: The collection explicitly includes the pre-pandemic panel. | concept_edges[5] | 0.95 | the tagger's words |
| concept_edges | samples from individuals aged at least 18 years with laboratory-confirmed SARS-CoV-2 infection —part-of→ blood (plasma and serum) sample collection the relation itself is separately evidenced · strength of the relation: 1 · note: The collection explicitly includes the infection-confirmed panel. | concept_edges[6] | 0.95 | the tagger's words |
| concept_edges | Readouts —operationalised-as→ fluorescent signal the relation itself is separately evidenced · strength of the relation: 1 · note: The readouts are measured as fluorescent signal before calibration. also in: Readouts · fluorescent signal | concept_edges[7] | 0.95 | the tagger's words |
| concept_edges | symptom severity —operationalised-as→ asymptomatic, mild, severe, critical or death the relation itself is separately evidenced · strength of the relation: 1 · note: The paper explicitly defines severity by these WHO-aligned categories. | concept_edges[8] | 0.95 | the tagger's words |
| concept_edges | specificity —operationalised-as→ known negative pre-pandemic samples the relation itself is separately evidenced · strength of the relation: 1 · note: Specificity is assessed with the pre-pandemic negative panel. also in: specificity · known negative pre-pandemic samples | concept_edges[9] | 0.95 | the tagger's words |
| concept_edges | sensitivity —operationalised-as→ known positive samples from patients who had previously had laboratory-confirmed SARS–CoV–2 infection the relation itself is separately evidenced · strength of the relation: 1 · note: Sensitivity is assessed with the laboratory-confirmed positive panel. | concept_edges[10] | 0.95 | the tagger's words |
| concept_edges | first 3–4 weeks after symptom onset —causes→ Antibody responses status: observational temporal trajectory across assays, not an intervention · direction on the measure: increases · the relation itself is separately evidenced · strength of the relation: 1 · status (stated by the tagger, no quotation): observational temporal trajectory across assays, not an intervention · note: Antibody responses increased over this post-symptom period.
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but antibody responses rose OVER the first weeks; a time window as the cause is loose. | concept_edges[11] | 0.9 | the tagger's words |
| content_tags | THERE IS NOTHING TO MEASURE THE INSTRUMENTS AGAINST, AND THE PAPER SAYS SO: `no clear gold standard exists ... RT-PCR positivity is a proxy ... could either reflect antibody test performance, or alternatively` four other things, one of which is A FALSE-POSITIVE RT-PCR TEST | content_tags.findings[0] | 0.95 | the tagger's words |
| content_tags | all assays reached at least 98% sensitivity with thresholds optimised for at least 98% specificity on samples 30 or more days after symptom onset | content_tags.findings[1] | 0.95 | the tagger's words |
| content_tags | and the authors keep it honest: `any post-hoc changes to thresholds would require repeat verification of sensitivity and specificity` | content_tags.findings[2] | 0.95 | the tagger's words |
| content_tags | THE COST OF THE CHOICE, IN PEOPLE: `at a 10% seroprevalence, the Siemens assay would generate an estimated 2800 total errors per million tests, versus the DiaSorin assay at 13 700` | content_tags.findings[3] | 0.95 | the tagger's words |
| content_tags | prior Abbott sensitivity estimates ranged from 93.4% to 100% (70-680 samples), while DiaSorin had a single 94.4% estimate on 18 samples | content_tags.findings[4] | 0.95 | the tagger's words |
| content_tags | AND NOBODY HAD RUN THEM TOGETHER: `No published studies assessed all four commercial platforms using the same sample sets` | content_tags.findings[5] | 0.95 | the tagger's words |
| content_tags | ONE INSTRUMENT WAS DROPPED BEFORE THE COMPARISON BEGAN: `the sensitivity of the EUROIMMUN assay has been reported as being approximately 90%. We therefore replaced this with the SARS-CoV-2 Total assay (Siemens)` | content_tags.findings[6] | 0.95 | the tagger's words |
| content_tags | AND THE ASSESSMENT CRITERIA THEMSELVES HAD BEEN RELAXED: `leading to a relaxation of typical assessment criteria (eg, the US Food and Drug Administration's [FDA] Emergency Use Authorization programme)` | content_tags.findings[7] | 0.95 | the tagger's words |
| content_tags | THE BEST INSTRUMENT CHANGES WHEN THE POPULATION DOES: `an optimised assay measuring antibodies to nucleocapsid protein might be most relevant for disease surveillance in the context of a widespread rollout of vaccines, which are mostly spike-protein based` | content_tags.findings[8] | 0.95 | the tagger's words |
| content_tags | AND WHO IS NOT IN THE SAMPLE IS STATED: `might under-represent some ethnic groups, did not include children, and captured insufficient clinical meta-data` | content_tags.findings[9] | 0.95 | the tagger's words |
| content_tags | AND THE PUBLISHED RECORD CARRIES ITS OWN CORRECTION: `This online publication has been corrected. The corrected version first appeared ... on November 25, 2020` - the PMC record we staged is version 2 | content_tags.findings[10] | 0.95 | the tagger's words |
| content_tags | `no clear gold standard exists against which to assess these antibody tests` - the substitute is `RT-PCR positivity ... a proxy for the expected presence of antibody` also in: gold standard | content_tags.definitions[0] | 0.95 | the tagger's words |
| content_tags | 2800 — the Siemens assay's projected total errors at 10% seroprevalence - the best of the five | content_tags.quantities[0] | 0.95 | the tagger's words |
| content_tags | 13 700 — the DiaSorin assay's projected total errors at the same seroprevalence - a five-fold gap from a choice of platform | content_tags.quantities[1] | 0.95 | the tagger's words |
| content_tags | 18 — the size of the only prior sensitivity estimate for the DiaSorin assay in the literature the authors could find | content_tags.quantities[2] | 0.95 | the tagger's words |
| content_tags | 98 — the sensitivity and specificity every one of the five assays reaches once the threshold is chosen for it rather than by the manufacturer | content_tags.quantities[3] | 0.95 | the tagger's words |
| content_tags | the Abbott assay's sensitivity at the manufacturer's threshold - the lowest of the five, and above 98% once the threshold is moved also in: 92·7% (95% CI 90·2–94·8) | content_tags.statistics[0] | 0.95 | the tagger's words |
| content_tags | the proxy standing in for an absent gold standard, with four named ways it can disagree with an antibody result without either being wrong also in: RT-PCR positivity | content_tags.variables[0] | 0.95 | the tagger's words |
Tags that could not be placed in the text:
What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.