Study Population, Datasets, and Variables

The training dataset, described in detail elsewhere (22), encompasses all patients suspected of having EVD (EVD-suspected patients) in Ituri and North Kivu provinces, Democratic Republic of the Congo (DRC). Data were collected during August 1, 2018–August 28, 2019, from 30 different Ebola treatment or transit centers and small, decentralized isolation units. Variables of interest for prediction of infection included time-to-presentation, age, 34 clinical variables, final GeneXpert (Cepheid, https://www.cepheid.com) RT-PCR status, and 4 possible exposure histories: contact with a known EVD-positive person, attendance at any funeral, health facility consultation for any reason, and consultation with an informal health practitioner for any reason. Following methods previously described (22), we considered 2 patient groups based on time-to-presentation (short vs. long), separated by a threshold between day 2 and day 3 after symptom onset (with symptom onset self-reported by patients or their relatives during in-depth epidemiologic investigations by trained investigators and clinical teams). Newly EVD-suspected cases occurring during August 15–November 28, 2019, comprised the testing dataset, which consisted of 14,346 patients, among whom 319 (2.2%) were confirmed EVD-positive by RT-PCR (Appendix).

Development of the Triage Algorithm

The diagnostic performance of predictors is reported from the training dataset according to multivariate logistic regression diagnostics (22), enabling a first selection of main predictors. The association of predictors with infection was either positive or negative and could vary by time-to-presentation. We incorporated 2 components into the algorithm: a prioritization rule for variables highly predictive of infection, and an EVD prediction score based on other variables having strong positive or negative associations with infection, also considering time-to-presentation. For this second component, we calculated individual scores from the regression diagnostics, based on β coefficients. We used a variable selection process to evaluate the performance of different sets of predictors with different ranges of individual scores. We evaluated versions of the algorithm on 30 bootstrapped samples of the training dataset and compared classification performance (area under the receiver operating characteristic [AUROC] curve). Among algorithm versions offering sufficient performance, we chose simplicity of use as the criterion to select the final version.

External Validation of the Triage Algorithm

For each patient from the testing dataset, and irrespective of the presence of priority variables, we calculated the EVD prediction score by summing individual scores of variables present. When the yes/no value of a scoring predictor was missing (20.2% of patients had >1 scoring variable missing), we assigned an individual score of zero (for that variable), assuming the said predictor was likely absent. We excluded patients missing time-to-presentation (589 patients, 4.1%) from this external validation.

We evaluated classification performance by using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), likelihood ratio of positive (LR+) tests, likelihood ratio of negative (LR–) tests, and AUROC to predict an EVD-positive diagnosis by RT-PCR (reference standard). Because of missing data among the 4 priority variables in the testing dataset, we considered 3 strata of patients: prioritized (high-risk) if any of the 4 priority variables were present (“Yes”); not prioritized if all of these 4 variables were absent (“No”); and unknown when none of the 4 variables had “Yes” but >1 was missing.

Prospective Evaluation

A prospective study evaluated this triage tool in real-life conditions. It was implemented as successive substudies, enabling a target sample size of 65 EVD-positive patients (to estimate sensitivity and specificity with 6% precision) (23). We present interim findings from 2 substudies during the 10th (Eastern Democratic Republic of the Congo [DRC], 2020) and 11th (Equateur, DRC, 2020) Ebola epidemics. Our objectives were to evaluate the performance of the developed algorithm in classifying EVD risk among patients matching the Ebola case definition, with GeneXpert RT-PCR as the reference diagnosis, and to assess ease of use by healthcare workers. Participants were patients validated by response teams as matching the Ebola case definition and consenting to be included in the study. Study sites were health facilities where the study was implemented (8 facilities during the 10th DRC epidemic and 9 facilities during the 11th DRC epidemic).

Before study start, healthcare workers (HCWs) responsible for triage received a 4-hour training session on study ethics, the informed consent process, study procedures (including history-taking and clinical evaluation), investigation of possible exposure histories, the triage algorithm, and study forms. A supervisory staff visited these HCWs a minimum of 3 times following training. HCWs recorded as “present” any sign or symptom occurring at any time since disease onset (even if that symptom had disappeared by the time of evaluation). Healthcare staff recorded exposure histories in the 21 days before disease onset. HCWs evaluated all study participants >1 time, immediately after inclusion, and evaluated some patients again, 24 hours later. The risk categorization did not change the procedures of isolation and testing.

After paper-based data collection, data clerks double entered data in the REDCap electronic data capture tool (https://www.project-redcap.org). We resolved any discrepancies before analyses. We computed concordance between in silico–computed and HCW-computed time-to-presentation, EVD prediction score, and EVD risk category as indicators of quality and comprehension by HCWs. We performed all analyses by using R statistical software (24).

Diagnostic Performance of Individual Predictors (Appendix Table 1)

In univariate analyses, the predictors offering the best sensitivities were asthenia (80.6%, 95% CI 78.8%–82.4%), anorexia (70.2%, 95% CI 68.0%–72.2%), and being a contact of an EVD case-patient (65.4%, 95% CI 63.1%–67.7%). Taken individually, other predictors, such as fever, had lower sensitivities, approaching or below 50%. Being a contact of an EVD case-patient was the most sensitive predictor for short time-to-presentation (80.2%, 95% CI 76.3%–83.7%), followed by asthenia (70.0%, 95% CI 65.9%–73.9%). Being a contact of an EVD case-patient had a specificity of 83.9% (95% CI 83.4%–84.4%), and other exposure histories had specificities of 90%–99%. In contrast, predictors having the highest sensitivities had lower specificities individually: asthenia at 31.2% (95% CI 30.6%–31.8%) and anorexia at 36.7% (95% CI 36.0%–37.3%).

Because of the low prevalence of EVD in the study population (4%), NPVs were high for all predictors (≥90%) but were the highest for being a contact of an EVD case-patient (96.7%, 95% CI 96.4%–97.0%), having attended a funeral (94.7%, 95% CI 94.4%–95.0%), and having asthenia (94.9%, 95% CI 94.3%–95.4%). Conversely, PPVs were low in general but were highest for bleeding at an injection site (70.6%, 95% CI 56.2%–82.5%), bleeding gums (43.7%, 95% CI 35.8%–51.8%), conjunctivitis (24.6%, 95% CI 21.4%–28.0%), being a contact of an EVD case patient (25.2%, 95% CI 23.9%–26.5%), and having attended a funeral (26.7%, 95% CI 24.9%–28.7%).

Two signs offered high (>8) LR+, bleeding at an injection site (25, 95% CI 13.7–45.5) and bleeding gums (8.1, 95% CI 5.9–11.0), although only for the long time-to-presentation group. Being a contact of an EVD case patient (4.1, 95% CI 3.9–4.3) and having attended a funeral (4.6, 95% CI 4.3–5.0) had lower LR+ (≈4), but those variables were relatively constant regardless of time-to-presentation. We noted negative predictors having LR+ <0.3 (arbitrary threshold) only in the short time-to-presentation group: epistaxis (0.3, 95% CI 0.1–0.8) and melena (0.3, 95% CI 0.1–0.6). Other negative predictors (i.e., abdominal pain, diarrhea for short time-to-presentation, cough, headache) had LR+ >0.5.

Choice of Predictors

Based on these findings and on those from prior work (22), we considered persons at high-risk (prioritization rule) to be those demonstrating any of the following 4 variables: 1) being a contact of an EVD case patient; 2) bleeding at the injection site; 3) having bleeding gums; and 4) having had contact with an informal healer (outside the health pyramid; eg, a private nurse or traditional healer). This 4th characteristic was not strongly associated with EVD in the database (reporting bias) but strongly linked with superspreading events in investigational reports (data not shown).

To classify patients not demonstrating any of the 4 predictors, we developed an EVD-prediction score based on the 2 described time-to-presentation groups and the remaining predictors having sufficient diagnostic performance and prevalence in >1 of the 2 time-to-presentation periods. Overall, AUROC resulting from application of various sets of predictors and individual scores on bootstrapped samples was 65.3%–73.9% for short time-to-presentation and 70.6%–74.4% for long time-to-presentation (Appendix). Using the new algorithm version 4.2 across 13 variables, we obtained the best compromise between performance and ease of use, with a +1 individual score assigned for odds ratios (ORs) >1, a −1 individual score for ORs <1, and a nil score for ORs of 1 or ≈1. The resulting AUROC of this scoring component version 4.2 was 71.4% (95% CI 69.8%–72.9%) for short time-to-presentation and 73.3% (95% CI 72.3%–74.4%) for long time-to-presentation (Appendix Tables 2–4).

Choice of 2 Thresholds

Figure 1

Figure 1. Performance of a rapid decision algorithm for patient triage during Ebola outbreaks (version 4.2, Ebola virus disease [EVD] prediction score only) for different decision thresholds to predict Ebola infection in...

Figure 2

Figure 2. Classification performance (area under the receiver operating characteristic curve) of a rapid decision algorithm for patient triage during Ebola outbreaks a population of EVD-suspected patients in Democratic Republic of the...

We evaluated only performances of the EVD-prediction score (excluding priority variables) on the testing population (Table 1; Figure 1). The AUROC was 68.3% (95% CI 67.9%–68.7%) (Figure 2). Because of the low prevalence of EVD, the NPV was consistently in the range of 98% to 100%, regardless of the value used as a threshold. Regarding negative EVD prediction scores, we anticipated the false-negative rate of <1%, given the situation of low EVD prevalence. However, according to our projections, we surmised that a slight decrease in NPV could also be anticipated in higher-prevalence settings (NPV = 92% for negative EVD-prediction scores in a setting with a confirmation rate of 20% [data not shown]). Conversely, the PPV was low in the lowest values of the prediction score but showed an inflection point from a threshold ≥3 when the PPV reached 10% (Figure 1). For the same threshold, however, the PPV could be expected to rise above 60% if prevalence increased to 20% (data not shown).

Likelihood ratios represent the multiplicative factor converting the pretest probability into a posttest probability of infection. They do not vary with prevalence and can be used at the individual level. Three thresholds (≥3, ≥4, ≥5) provided sufficient LR+, and 2 other thresholds (≤−2, ≤−1) provided adequate LR– (Table 1).

Of note, when inspecting separately the performances for the 2 time-to-presentation groups, sensitivity was slightly higher for long time-to-presentation than for short time-to-presentation. This finding would lead to lower performance detecting EVD-positive cases shortly after disease onset. Both LR+ and LR– also performed better for long time-to-presentation.

Figure 3

Figure 3. Final triage algorithm (version 4.2) for the evaluation of the likelihood of Ebola infection among EVD-suspected patients in Democratic Republic of the Congo, during epidemics in 2018–2019. Three risk categories...

Given those expected performances and numbers of patients in each category, we determined the most suitable thresholds (0; 2) to be for the intermediate-risk category (Figure 3). For the lowest threshold, we favored the combination of an optimal size of risk categories with the highest sensitivity to reduce the false-negative rate.

Positivity Rates and Viral Loads

Figure 4

Figure 4. EVD confirmation rates (blue line) and number of patients (bars) classified by EVD prediction score obtained by a rapid decision algorithm for patient triage during Ebola outbreaks used in a...

Figure 5

Figure 5. Distribution of Ebola GP cycle threshold Ct values among EVD-positive patients and averages of Ct values (orange line) by EVD prediction score obtained by a rapid decision algorithm for patient...

When applying the scoring component of our new version 4.2 algorithm to the testing population, EVD prediction scores were distributed around a mode of zero, and we observed a slight shift to higher values for long time-to-presentation. As expected, EVD confirmation rates were associated with EVD prediction scores (Table 2, Figure 4) and their distribution confirmed the choice of the 2 thresholds. From the score +1, this rate reached or exceeded the average in all 3 subgroups: prioritized, unknown (missing data for some priority variables), and nonprioritized patients. A clear trend of increasing viral load with increasing EVD prediction scores (Figure 5) suggested that these parameters reflected not only the likelihood of infection, but also disease severity.

Performance of Priority Variables

In the testing population, 1,302 patients (9.1%) had >1 of the 4 priority variables, 9,411 (65.6%) had none, and 3,633 (25.3%) had none but with some missing information. We looked at the prevalence of each of the 4 characteristics among EVD-positive and -negative patients, in particular the much higher frequency of these characteristics among the EVD-positive persons (Appendix Table 5). Adding this prioritization rule, we then multiplied the size of the high-risk category by about 4 to 8 (compared with the EVD prediction score only), depending on the age group considered (Table 3), with relatively steady confirmation rates. We noted, however, that sensitivities of the lower and upper thresholds in all age groups were significantly improved: lower threshold was 88.1% (95% CI 84.0%–91.4%) and upper threshold 13.8% (95% CI 10.2%–18.1%) without the prioritization rule; and lower threshold 91.2% (95% CI 88.1%–94.3%) and upper threshold 56.7% (95% CI 51.3%–62.2%) with the prioritization rule. Of note, NPV of the low-risk category was highest (100%) in children 0–5 years of age. Of the testing population, the low-risk category represented 30.1% (95% CI 29.4%–30.9%), the intermediate-risk category represented 58.8% (95% CI 57.9%–59.6%), and the high-risk category represented 11.1% (95% CI 10.6%–11.6%).

Comparison With Other Classifiers

We compared performances of our newly developed algorithm (version 4.2) with those from previous work (15,17,19) based on our testing population. Because of the prioritization rule, the sensitivity of our algorithm could not drop below 50% in lowest scores and, conversely, its specificity was limited to 90% in highest scores. This limitation prevented a comparison of our AUROC results with those of other tools. All algorithms showed PPV increasing with prediction scores except Oza’s version (Appendix Figure 1). In their highest prediction scores, Hartley and Levine’s versions seemed to have suitable PPV and LR+ for late presenters but not for early presenters. Compared with other developed tools, our new algorithm also seemed to show more favorable results in terms of NPV and LR–.

Interim Results of the Prospective Study

We investigated data obtained from 2,652 EVD-suspected patients involved in the 10th (N = 2,206) and 11th (N = 446) DRC Ebola epidemics. We also compiled information in 2 substudies, where 102 patients underwent a second evaluation after 24 hours. After excluding patients with missing data, we focused our analysis on a total of 2,695 evaluations (Appendix Table 6, Figure 2).

The triage tool procedure appeared understandable to health personnel; however, some aspects of the history-taking process required further instruction. Because HCWs often were not familiar with ongoing chains of transmission, nor with step-by-step approaches for such investigations, supervision appeared crucial to improve the collection of variables, especially exposure histories to potential Ebola case-patients. Tool assessment showed good concordance; 95.8% of evaluation forms had correct calculation of time-to-presentation, 98.5% had a correct choice of the set of individual predictors, and 95.3% showed EVD scores correctly computed. Quality of patient evaluation at triage required good engagement between health personnel and the supervisory study team.

Eight EVD-positive patients were identified in the study population, 2 of whom had received Ebola vaccination (Table 4). EVD prediction scores were above +2 in 2 of the 8 cases, and detection of some of the 4 priority variables reclassified 3 of the 8 cases into the high-risk category. We therefore classified 5 of the 8 cases as high-risk and the other 3 as intermediate-risk. According to the collected variables, the EVD prediction scores and risk categories calculated by HCWs were correct; however, a review of narratives from investigation teams revealed difficulties in detecting some exposure histories.

Considering all included EVD-suspected patients, and as also seen with the retrospective data, we observed an increase in the likelihood of infection by EVD risk category (Table 5), and the relative sizes of the 3 risk categories were similar to those obtained with the testing dataset. Of note, at this interim stage, the low-risk category (38.9% of evaluated patients, 95% CI 37.1%–40.8%) included no EVD-positive cases.