Selection of Pathogens

We included all pathogens transmitted commonly through food or water that were examined by Scallan et al. (2) and Collier et al. (13) except those for which the only syndrome of interest was considered to have >95% foodborne transmission (e.g., Listeria monocytogenes, Clostridium botulinum); we added 3 free-living amoebae (2,13). For some pathogens, subdivisions into categories by serotype, patient age, or clinical manifestations of interest were included because transmission pathways were assumed to be different. For example, for Salmonella, the 5 most common serotypes were included along with 2 groups of rarer serotypes based on a ranking of their coefficients of variation (CVs) calculated from the patients’ ages, sexes, states of residence, and the year and month specimens were obtained (group 1, lowest CVs; group 2, highest CVs) as described by Boore et al. (14). This compilation resulted in a total of 33 pathogens and 47 target questions, or categories, for estimation. The 47 target questions were grouped into 15 panels on the basis of similarities between pathogen microbiology and ecology (Table 1).

Transmission Pathway Definitions

We used definitions for 5 major pathways that were mutually exclusive and comprehensive (i.e., covering 100% of transmission modes) and that reflect those used by CDC for outbreak surveillance (15,16; Tables 2, 3). We defined 3 mutually exclusive waterborne subpathways (recreational water, drinking water, and nonrecreational nondrinking water) that were comprehensive (i.e., all waterborne pathway transmission fell into 1 of the 3 subpathways). We also defined and elicited 1 foodborne (food handler-related) and 2 environmental (presumed animal associated, presumed person-to-person) subpathways that accounted for only a portion of transmission within their main pathway. We calculated the unelicited proportion remaining of their respective main pathways during analysis and assigned it to the subpathways other foodborne and other environmental. For all transmission pathways, we defined the point of attribution as the point of exposure (i.e., the event during which a person ingested, or was otherwise exposed to, the pathogen).

Expert Identification and Selection

Figure 1

Figure 1. Expert selection process for study of attribution of illnesses transmitted by food and water to comprehensive transmission pathways using structured expert judgment, United States, 2017.

We identified 182 experts representing a range of scientific backgrounds (e.g., epidemiologists, laboratory scientists, and environmental engineers from government, academia, nongovernmental organizations, and industry) on the basis of publication records, experience, expertise, or previous participation in source attribution studies. We contacted the experts directly and invited them to apply for participation (Figure 1). Fifty-eight returned a curriculum vitae and publication record and completed a questionnaire about their professional interest, knowledge, and experience for each of the 33 pathogens using a 4-level Likert scale (high, medium, low, or none) by the requested deadline. We asked experts to suggest additional experts to be considered; the 3 who were suggested were also invited.

Assignment to Panels

We evaluated expert applications based on area of expertise, education, work history, professional interest, experience, and knowledge of the individual pathogens in this study. Publication record was not used to determine eligibility because it could have led to elimination of qualified experts who do not publish frequently. We used maximum bipartite matching in R version 3.3.1 with the igraph package version 1.0.1 to assign experts to panels based on their curricula vitae, publication records, and questionnaire responses (17,18). Final assignment ensured that experts were not on pathogen panels for which they reported none or low experience. Individual experts were on panels for <15 pathogens (Appendix 1).

Calibration Questions

The study administrators used unpublished data to develop calibration questions (Appendix 2). We developed 14 questions to evaluate the experts’ statistical accuracy and informativeness by probing the experts’ ability to provide reliable estimates under uncertainty. The subject domain of the questions aimed to represent expertise in public health surveillance of foodborne and waterborne diseases, food consumption patterns in the United States, and human exposure and occurrence data about pathogens in food, water, and the environment.

Target Questions

Target questions asked the proportion of illnesses transmitted through the 5 major pathways and 6 subpathways for all study pathogens. Study administrators blocked transmission pathways and subpathways for some pathogens based on their microbiology and ecology (Table 4). We created individualized Microsoft Excel version 14.7.7 (http://www.microsoft.com) files with separate sheets for calibration questions, target questions for each assigned pathogen, and additional instructions for each expert. We included verification aids in the worksheets to assist the experts (Appendix 3).

Dry Run Exercise

We conducted a dry run exercise using video web conferencing to assess calibration questions, target question answer sheets, and expert training materials for completeness, clarity, and ease of use. Six persons from academia, state health departments, and CDC participated in this trial exercise, but not in the formal elicitation itself. We modified the elicitation materials based on feedback from this exercise.

Expert Orientation

Before the formal elicitation, experts attended a training webinar to learn definitions of transmission pathways, subpathways, and point of attribution. To ensure common understanding of the definitions, experts completed a 20-question review of knowledge after the webinar (Appendix 4).

We provided a background document summarizing current surveillance data, when available, and relevant research findings for each pathogen. The document contained links to selected research articles. Experts were encouraged to use any data they felt were informative to make their estimates; they were not limited to only this document.

Calibration Questions

Experts were not expected to know true values precisely and provided low (5th percentile), median (50th percentile), and high (95th percentile) estimates to represent their uncertainty on the answers provided to the calibration questions. Experts were not allowed access to any additional resources while answering the calibration questions and, after they had they had finished, they could not return to this section to change their responses.

Target Questions

After completion of the calibration questions, experts provided 5th, 50th, and 95th percentile estimates for the proportion of domestically acquired illnesses that are transmitted through each major pathway and subpathway annually for each pathogen and target question in each panel to which they were assigned. The experts were also asked to indicate if they did not agree with the pathways blocked by study administrators. One pathway, person-to-person transmission for Legionella spp., was unblocked based on this feedback, and experts provided this estimate with the others at the in-person elicitation. Experts could access resources and discuss them with colleagues, if desired. However, we emphasized that the final estimates should represent the expert’s individual responses, not a group consensus.

Re-Elicitation

After the in-person elicitation was completed, we determined that re-elicitation for some pathogens was necessary. More granular detail was needed beyond the single estimate for Pseudomonas, so estimates were re-elicited for otitis externa, septicemia, and pneumonia. Based on feedback we received during the elicitation, we re-elicited estimates for non–acute gastrointestinal infections (non-AGI) for nontoxigenic Vibrio cholerae, V. parahaemolyticus, V. vulnificus, and V. alginolyticus. Experts were provided with feedback with updated surveillance data and given the opportunity to adjust their original estimates if new data led them to reconsider their previous estimates (Figure 1). The re-elicitations were completed through follow-up emails and web conferences.

Data Analysis

We analyzed data using EXCALIBUR (19). We combined all individual expert assessments by linear pooling into a single uncertainty assessment for each target question (11). For equal-based weighting, all experts’ assessments contributed to the combined uncertainty assessment evenly. We computed performance-based weighting by combining the statistical accuracy and information scores of experts in each panel. The weighted combination of experts is referred to as the decision maker. We used the item weight decision maker because this calculates and applies weights per individual target question rather than for all questions an expert answered. We performed optimization to determine the threshold by which an expert’s responses would be included in the final estimate or not. This was done separately per expert for each panel, based on each expert’s statistical accuracy score (12).

We performed a subgroup analysis to determine whether separate schools of thought existed as a result of experts’ self-identified background (categorized as mainly foodborne, mainly waterborne, or both). This analysis was completed by 2 independent reviewers who analyzed EXCALIBUR panel outputs for each target question to determine whether wide divergence existed among individual responses.

We normalized random samples from the weighted distributions for major transmission pathways and waterborne subpathways such that on each sample the values across pathways summed to 1. This process was done by resampling the cumulative distribution functions generated by EXCALIBUR 5,000 times in R version 3.4.3 for each pathogen, while dividing all sampled values by the sum of their values per iteration. Point estimates and 95% uncertainty intervals (UIs) for each target question and pathway were produced. We performed robustness analysis and out-of-sample validation to assess the performance of the method and to evaluate the effect of individual experts and individual calibration questions on the final distribution (Appendix 5) (12).

Bacteria

Most of the pathogens in this study were bacteria; they encompassed 35 of the 47 target questions. More than half of transmission (>50%) was attributed to the foodborne pathway for Campylobacter spp.; enterotoxigenic Escherichia coli; Shiga toxin-producing Escherichia coli (STEC) O157; other diarrheagenic E. coli; Mycobacterium bovis; nontyphoidal Salmonella enterica (all ages and serotypes); S. enterica serotypes Enteritidis, I 4,[5],12:i:-, Javiana, Newport, Typhimurium, and group 1 serotypes; Vibrio alginolyticus; V. cholerae nontoxigenic; V. parahaemolyticus; Vibrio spp., other AGI; Vibrio spp, other non-AGI; and Yersinia enterocolitica. In addition, Legionella spp.; nontuberculous Mycobacterium spp.; Pseudomonas spp., otitis externa; invasive Staphylococcus aureus; V. alginolyticus, non-AGI; V. cholerae nontoxigenic, non-AGI; V. parahaemolyticus, non-AGI; and V. vulnificus were all estimated to have majority transmission from the waterborne pathway. Most transmission for Shigella spp. and group A Streptococcus were estimated to be through person-to-person transmission. No bacterial pathogen had majority transmission through animal contact. Pseudomonas spp. septicemia was attributed primarily to the environmental pathway.

Protozoa

Cyclospora cayetanensis was the only protozoan estimated to have majority transmission through the foodborne pathway. Acanthamoeba spp. and Naegleria fowleri both had >80% transmission attributed to the waterborne pathway, and 54% (UI 5%–95%) of Balamuthia mandrillaris infections were estimated to occur through waterborne transmission. No protozoa had majority person-to-person or environmental transmission. Waterborne transmission was estimated at 43% (UI 17%–73%) for Cryptosporidium spp. and 44% (UI 16%–78%) for Giardia spp. Among all pathogens, Toxoplasma gondii had the highest attribution to animal contact transmission, 58% (UI 24%–86%).

Viruses

Most transmission for astrovirus, norovirus, rotavirus, and sapovirus was attributed to the person-to-person pathway. Hepatitis A virus was estimated to have the highest proportion of illness transmitted by the foodborne pathway at 42% (UI 9%–78%). Of this, 48% (UI 2%–93%) was considered food handler related. Of foodborne transmission, 50%–71% was estimated to be food handler related for astrovirus, norovirus, and sapovirus. For all viruses, 67%–88% of environmental transmission was attributed to the subpathway of presumed person-to-person transmission.