Volume 10, Number 1—January 2004
Research
Evaluating Detection and Diagnostic Decision Support Systems for Bioterrorism Response
Dena M. Bravata*†
, Vandana Sundaram*†‡, Kathryn M. McDonald*†, Wendy M. Smith*†, Herbert Szeto*†§, Mark D. Schleinitz¶#, and Douglas K. Owens*†‡
, Vandana Sundaram*†‡, Kathryn M. McDonald*†, Wendy M. Smith*†, Herbert Szeto*†§, Mark D. Schleinitz¶#, and Douglas K. Owens*†‡Table 3
Evaluation data for diagnostic decision support systems for bioterrorism-related illnessa
| System name | Purpose | Evaluation datab |
|---|---|---|
| Clinical decision support system for detection and respiratory isolation of tuberculosis patients (21) | To automate the detection and respiratory isolation of patients with positive cultures and chest x-rays suspicious for TB. | In a retrospective analysis, the system increased the proportion of appropriate TB isolations in inpatients from 51% to 75% but falsely recommended isolation of 27 of 171 patients. In a prospective analysis, the system correctly identified 30 of 43 of patients with TB but not identify 21 of these patients (false-negatives). However, the decision support system identified 4 patients not identified by the clinicians (21). |
| Columbia–Presbyterian Medical Center Natural Language Processor (22) | To automate the identification of 6 pulmonary diseases (including pneumonia) through analysis of radiology reports. | The system had a sensitivity of 81% (95% confidence interval [CI] 73% to 87%) and a specificity of 98% (95% CI 97% to 99%) compared to physicians who had an average sensitivity of 85% and specificity of 98% (22). |
| Computer Program for Diagnosing and Teaching Geographic Medicine (23) | To provide a differential diagnosis of infectious diseases matched to 22 clinical parameters for a patient; also to provide general information about infectious diseases, anti-infective agents, and vaccines. | The computer program correctly identified 75% (222 of 295) of the actual cases and 64% (128 of 200) of the hypothetical cases of patients with infectious diseases (23). The clinical diagnosis was included in the computer differential diagnosis list in 94.7% of cases. Among the cases included in this evaluation, several were for bioterrorism diseases (23). |
| DERMIS (24,25) | To provide a differential diagnosis of skin lesions. | The system correctly diagnosed lesions 51% to 80% of the time and included the correct diagnosis among its top 3 choices 70% to 95% of the time (out of a total of 5,203 cases) (24,25). The system was more accurate for dermatologist users than general practitioners. |
| Dxplain (26) | To provide a differential diagnosis based on clinician-entered signs and symptoms. The system includes descriptions and findings for potential bioterrorism agents, and is updated weekly to account for potential outbreaks. | In an evaluation of 103 consecutive internal medicine cases, Dxplain correctly identified the diagnosis in 73% of cases, with an average rank of 10.7 (the rank of a diagnosis refers to its position on the differential diagnosis—for example, the diagnosis with the greatest likelihood of being the actual disease is ranked first and the next most likely diagnosis is ranked second) (26). |
| Fuzzy logic program to predict source of bacterial infection (27) | To use age, blood type, gender, and race to predict the cause of bacterial infections. | The program was able to correctly classify 27 of 32 patients into 1 of 4 groups based on demographic data alone (27). |
| Global Infectious Disease and Epidemiology Network (GIDEON) (28) | To provide differential diagnoses for patients with diseases of infectious etiology. All potential bioterrorism agents as specified by CDC are included in the GIDEON knowledge base (28). | Whereas medical house officers listed the correct diagnosis first in their admission note 87% of the time (for 75 of 86 patients), GIDEON provided the correct diagnosis for 33% (28 of 86 patients) (28). |
| Iliad (and Medical HouseCall which is a system for consumers derived from Iliad) (29–31) | To provide a differential diagnosis based on clinician-entered signs and symptoms. The knowledge base is focused in internal medicine and was last updated in 1997. | In a multicenter evaluation, each of 33 users analyzed 9 diagnostically difficult cases. On average, Iliad included the correct diagnosis in its list of possible diagnoses for 4 of the 9 cases, and included the correct diagnosis within its top 6 diagnoses for 2 of the 9 cases. The differential diagnosis generated by Iliad is not dependent upon the level of training of the user (29–31). |
| Neural Network for Diagnosing Tuberculosis (32) | To predict active pulmonary TB (using clinical and radiographic information) so that patients may be appropriately isolated at the time of admission. | The neural network correctly identified 11 of 11 patients with active TB (100% sensitivity, 69% specificity) compared with clinicians who correctly diagnosed 7 of 11 patients (64% sensitivity, 79% specificity) (32). |
| PNEUMON-IA (33) | To diagnose community-acquired pneumonia from clinical, radiologic and laboratory data. | The decision support system correctly identified pneumonia in 4 of 10 cases, compared with between 3 and 6 cases for the clinician experts (33). |
| Quick Medical Reference (QMR) (34) | To provide a differential diagnosis based on clinician-entered signs and symptoms. | One prospective study used QMR to assist in the management of 31 patients for which the anticipated diagnoses were known to exist in the QMR knowledge base. In the 20 cases for which a diagnosis was ultimately made, QMR included the correct diagnosis in its differential in 17 cases (85%) and listed the correct diagnosis as most likely in 12 cases (60%) (34). |
| SymText (35,36) | To analyze radiology reports for specific clinical concepts such as identifying patients with pneumonia. | Average sensitivity and specificity for assessing the location and extension of pneumonia was 94% and 96% for physicians and 34% and 95% for SymText. In selecting patients who are eligible for the pneumonia guideline, the area under the ROC curves was 89.7% for SymText and 93.3% for physicians (35,36). |
| Texas Infectious Disease Diagnostic Decision Support System (37) | To provide a weighted differential diagnosis based on manually entered patient information. | The system was compared to a reference standard that missed the diagnosis of 98 of 342 cases of brucellosis. In 86 of the 98 patients, this system listed brucellosis in the top 5 diagnoses on the differential diagnosis list, and in 69 of these 98 patients, brucellosis was the only disease suggested by the system. The system missed the diagnosis in 12 of 98 patients. On average, without the system it took 17.9 days versus 4.5 days with the system to suspect the correct diagnosis (37). |
| University of Chicago – Artificial Neural Network for Interstitial Lung Disease (38) | To help radiologists differentiate among 11 interstitial lung diseases by using clinical parameters and radiographic findings to develop a differential diagnosis. | Areas under the ROC curve obtained with and without the system output were 0.911 and 0.826 (p < 0.0001), respectively (38). |
| University of Chicago – Computer Aided Diagnosis of Interstitial Lung Disease (39) | To aid in the detection of interstitial lung disease in digitized chest radiographs. | Areas under the ROC curve obtained with and without computer-aided diagnostic output were 0.970 and 0.948 (p = 0.0002), respectively (39). |
aTB, tuberculosis; CDC, Centers for Disease Control and Prevention; ROC, receiver-operating characteristic curve.
bWhere possible, we report sensitivity and specificity data (and highlight them in bold); if the published reports did not provide these values directly but did provide sufficient data for them to be calculated, we performed these calculations.
