Identification and Definition of Risk Elements

Leading influenza subject matter experts in human and animal health, along with laboratory workers, field epidemiologists, and risk modelers, were invited to a meeting in 2011 (Technical Appendix) and asked to provide input into the development of the IRAT. Participants were asked to provide and refine definitions of elements used by the IRAT to evaluate the potential risk of novel influenza viruses. To distinguish among novel influenza viruses, each element must be amenable to qualitative or quantitative evaluation, independent of other elements, and must not be redundant in what is being measured. Subject matter experts agreed on and defined 10 elements to be scored that would provide the necessary critical information. Each of the 10 elements falls into 1 of 3 categories: virus properties, attributes and immune response of the human population, and the ecology and epidemiology associated with the novel influenza virus (Table 1). These 10 elements are meant to capture the features of the virus, the population, and the ecology and epidemiology likely to affect whether or not a specific zoonotic virus can spread widely in humans. These elements are also meant to capture factors likely to affect public health during a pandemic caused by a specific virus.

Definitions required for each element included scoring criteria that characterize viruses as low, moderate, and high risk to assist the experts in estimating a score on a scale of 1–10. For example, the definition of the element “Antivirals and Treatment Options” states, “For the purposes of the risk assessment tool, antiviral susceptibility refers to the predicted or demonstrated efficacy of available antiviral agents against animal influenza viruses.” A low-risk score (1–3) for this element is defined as “no evidence of clinically relevant resistance to any of the antiviral drugs approved for human use (neuraminidase inhibitors and M2 blockers).” A moderate-risk score (4–7) is defined as “sensitive to all neuraminidase inhibitors but resistant to M2 blockers.” A high-risk score (8–10) for this element is defined as “resistant to one or more neuraminidase inhibitor antiviral drugs.” All 10 elements have definitions for low-, moderate-, and high-risk scores.

Weighting of Risk Elements

Because not all risk elements are equally useful in answering each risk question, a weighting system had to be determined before the final risk score for each virus was generated. A key requirement for creating a weighting system was minimizing bias regarding the weighting of input from each expert’s field of study relative to that of others and reducing the difficulty of eliciting relative weights from experts when multiple attributes are compared. A surrogate weighting method was selected for this purpose (10). First, the expert panel was surveyed to determine a consensus rank ordering of the 10 risk elements if each risk question was asked separately. When asked to rank the elements when the “emergence question was considered,” 27 of 29 subject matter experts agreed that evidence of human infections is the “most important” of the 10 elements regarding risk that a novel influenza virus can achieve sustained human-to-human transmission. The other 2 subject matter experts believed this element was “very important.” Second in importance was information regarding laboratory animal transmission studies; 26 subject matter experts thought this element was “most important” or “very important.” Third in importance was receptor binding information and population immunity. Of least importance for the experts was information regarding antiviral and treatment options and disease severity; 24 of the 29 subject matter experts rated this element as “not relevant” for the question about the risk that a novel influenza virus can achieve sustained human-to-human transmission.

The second risk question addresses the potential effect of a novel influenza virus on public health in the event that the virus achieved sustained human-to-human transmission (i.e., the impact question). Subject matter experts agreed that knowledge regarding disease severity ranked first (26 ranked this element as “most important” and 3 ranked it as “very important”), followed by information pertaining to population immunity (21 ranked this element as “most important” and 8 ranked it as “very important”), human infections, and antiviral and treatment options in descending order. Of least importance were the elements related to global distribution in animal species (16 ranked this element as “not relevant”) and infections in animals (11 ranked this element as “not relevant”). Although risk element definitions were the same for the emergence and impact questions, the rank order of the risk elements by experts differed for the 2 questions.

This rank ordering of risk elements for the 2 questions facilitated the calculation of surrogate weights by using the formula

where K is the number of elements and Wk is the calculated weight to be applied to each risk element in rank order (10). Thus, the weight for the top-ranked element of the 10 total elements was calculated as 0.2929, and the lowest-ranked element was assigned a weighting score of 0.001. By convention, the sum of all 10 weights in this example is 1.

Scoring

Using the framework of risk element definitions, scoring criteria, and rank-ordered surrogate weights, individual experts were asked to evaluate viruses by using a particular risk element within their field of expertise and to provide a point estimate score. The returned scores for each risk element were averaged and then multiplied by the corresponding surrogate weight, depending on the risk question being asked. A summation of the product of all 10 risk elements provided the final aggregate score for each virus. Tables 2 and 3 show results of subject matter experts’ scoring of 3 avian influenza viruses. The H1N1 subtype virus found in North American mallards was used as an example of an influenza virus that has not infected people and appears to be wholly avian in characteristics.

Addressing Uncertainty

Inherent in any risk assessment is the need to address uncertainty. The IRAT attempts to capture this factor by asking subject matter experts to characterize uncertainty in 2 ways. First, uncertainty is captured by each expert’s point estimate score on a scale of 1–10 (i.e., 1 representing the lowest level of risk and 10 representing the highest), along with an upper and lower boundary for their scores, thereby providing a range for their point scores. A second measure of uncertainty uses a confidence score from each expert; this score expresses the expert’s confidence in the available data used to make the point estimate. Finally, the subject matter experts were asked to provide a basis or justification for their risk scores. This justification could include key references and data along with professional observations and experiences. This approach was based on the assumption that subject matter experts using the same knowledge base will produce similar scores for each virus and narrow uncertainty ranges unless data are insufficient or the experts have a fundamental disagreement in the interpretation of available data. The upper and lower bounds of the score for each risk element can be used to determine the highest or lowest (respectively) risk score for a given virus. In addition, a sensitivity analysis wherein the rank order of 2 risk elements can be swapped and the risk score recalculated on the basis of the new weights can be used to show the effect of these changes on a virus’s aggregate score. The level of uncertainty is communicated in the final report of virus scores so that decision makers using the IRAT can use this information in their considerations. Even with these efforts to measure and communicate uncertainty, the numerical score produced by the IRAT may create a sense of precision that is unfounded. Anyone making decisions by using the IRAT scores needs to be reminded that the scores are semiquantitative.

H3N2v Virus

In July 2011, the Centers for Disease Control and Prevention (CDC; Atlanta, GA, USA) confirmed the first human case of an influenza A(H3N2) variant [A(H3N2)v] virus that had acquired the matrix gene from the A(H1N1)pdm09 virus. Eleven additional cases were reported during the rest of 2011, for a total of 12 confirmed cases in 5 states (11). A preliminary risk assessment using the IRAT was conducted on the basis of information gathered from these 12 cases and available laboratory data (Tables 2, 3). The H3N2v risk score for both the emergence and impact questions in December 2011 placed this virus in the moderate-risk category; the summary scores for both questions were >5.0.

Figure 2

Figure 2. Influenza Risk Assessment Tool scores for 4 influenza viruses on the basis of potential risk to achieve “sustained human-to-human transmission” (emergence) and potential risk “for significant impact on public health” (impact)....

In 2012, a total of 309 cases of A(H3N2)v virus were reported from 12 states. In most cases, the putative source was direct or indirect contact with swine (12), with no evidence of community transmission. Studies conducted to assess population immunity (13–15) provided additional information regarding age groups considered at risk for infection. In 2013, subject matter experts rescored the H3N2v virus (Figure 2) in light of these and other available data. Although 321 confirmed cases were reported over 2 years, the summary scores for the virus were lower than those for the previous scoring. The risk score for the emergence question dropped from 6.5 to 6.1, and the risk score for the impact question dropped from 5.4 to 4.5. This reduction in the scores was caused primarily by information about population immunity. This information indicated that persons at greatest risk for infection were children, especially those <10 years of age; those 20–50 years of age exhibited cross-reactive antibodies against this virus. In addition, the epidemiologic investigations of the 309 cases in 2012 showed that >90% of the patients were <18 years of age; of these cases, 16 hospitalizations and 1 death associated with A(H3N2)v virus were documented (16,17). These epidemiologic observations reinforced the population immunity data obtained in the laboratory and supported the virus’s decreased cumulative risk score from its risk score in 2011.

H7N9 Virus

On March 31, 2013, the China Health and Family Planning Commission notified the World Health Organization of 3 cases of human infection with influenza A(H7N9) virus (4). As of April 8, 2013, the China World Health Organization Collaborating Centre for Reference and Research on Influenza, the China OIE (World Organisation for Animal Health) Reference Laboratory for avian influenza, and a provincial public health laboratory in China had identified and analyzed 7 virus isolates. The complete viral genome sequences of 6 isolates were deposited in a publicly accessible influenza database, and additional human cases continued to be reported.

In April 2013, subject matter experts used the IRAT and its 10 individual risk elements to assess the potential pandemic risk for humans posed by the novel A(H7N9) virus. To address the risk of the virus to achieve human-to-human transmission (emergence question), information about the virus’s ability to transmit in laboratory animals was critical. This information was not available in mid-April 2013. The absence of information for this and other elements affected the scoring of the risk elements. The scoring for the virus’s potential for sustained human-to-human transmission was in the moderate-risk category, approaching 5.0 on a scale of 1–10 (Figure 2). The moderate-risk score perhaps emphasized missing information in >1 high-ranking element (Transmission in Laboratory Animals). For the impact question, the risk score was in the higher end of the moderate-risk range (4–7), almost 6.0. Cautionary notes regarding interpretation of the scoring in the face of data gaps were included in the subsequent report.

A month later, in mid-May, a second scoring was conducted among CDC subject matter experts (Table 4). At this time, preliminary results of studies assessing the virus’s ability to transmit between animals in a laboratory setting were available. This information and updated information pertaining to the other elements were incorporated into the evaluation process. The risk score for the virus to achieve sustained human-to-human transmission was 6.2. This score was higher than the previous score based on the preliminary evaluation in April, although both scores were still in the moderate-risk range (4–7). The risk score for the virus’s potential to impact public health if it were to achieve sustained human-to-human transmission is 7.0, which falls in the upper limit of the moderate-risk range and which is higher than the preliminary score of ≈6.0 in mid-April (Figure 2).

During both rounds of scoring, the subject matter experts were asked to grade their level of confidence in the available data as it applied to their scoring. They generally expressed a greater confidence in their risk scores in May. For most elements, the risk scores differed little from those in the first round. However, in May, preliminary data were available for the risk element Transmission in Laboratory Animals and informed the risk score for this element. Although the 2 scorings took place only 1 month apart, this change illustrates how a risk score can increase after availability of new data and emphasizes the necessity of reevaluating risk scores when new data become available.