WHO developed a comprehensive methodology (3) to ensure the list of the R&D Blueprint prioritized diseases best reflects targeted global health needs and focuses on the most pressing threats. The approach taken drew heavily on established best practice (5–7) and is based on practical national and regional experiences in compiling similar lists (8–14). This approach also specifically addressed criticism of pre–R&D Blueprint attempts by WHO to prioritize diseases by developing tools for assessing confidence in the results generated and addressing potential biases (5).

Disease prioritization is not a straightforward task and requires a defined set of criteria on which to base prioritization (7). These criteria can be qualitative, intangible, or subjective, changing for different stakeholders (15). The criteria can also be interdependent, complicating separate assessment (16). For instance, the case-fatality rate of a disease has a social effect, which in turn has an economic effect. Given the complexity and the challenges of disease prioritization, ensuring the process is transparent and reproducible is important (5,7,17).

Recent disease prioritization methods were summarized in a 2015 review by the European Centre for Disease Prevention and Control (ECDC) (5), which extrapolated a series of best practices. Several subsequent studies were also identified (Technical Appendix 3). Past disease prioritization studies have been conducted for different purposes, such as communicable diseases surveillance (18,19), biosecurity (20), and resource allocation (21,22), and have covered disease in humans, livestock, or wildlife. Many studies were conducted primarily at national (8–10,19,21–36) and regional (11–14,16,20,37–43) levels (e.g., Europe and North America) but rarely at a global level (44,45). None of the disease prioritization exercises matched the aims of the R&D Blueprint, its public health focus, and its global reach; thus, WHO needed to develop its own methodology.

Several different disease prioritization methods exist (5), including Delphi processes (38,40), multicriteria decision analysis (MCDA) (14,26,28,36,46), H-index (42,43), questionnaires (11,13,22), and qualitative algorithms (47,48). Each method has its own strengths, weaknesses, and context-dependent utility, but 3 methods most closely matched the requirements of the R&D Blueprint (5): 1) a semiquantitative Delphi process to narrow the list of diseases under consideration; 2) MCDA to rank the remaining diseases (Technical Appendix 4); and 3) questionnaires in the form of online survey tools to standardize information gathering from participating experts.

The resulting methodology was developed over a year-long process, involving informal consultations, internal and external expertise, and guidance from the R&D Blueprint Scientific Advisory Group (4). Methods and tools were subsequently reviewed and validated by an external group of experts (49) and used in the review of the list of priority diseases in January 2017 (50).

Disease Scoring

Figure 1

Figure 1. Example screenshot of tool developed in R Shiny (https://shiny.rstudio.com) using the slide bar function to compare candidate diseases for each criteria and subcriteria considered in the development of the World...

To rank the short list of diseases, an online survey tool was designed by using the slide-bar function of R Shiny (https://shiny.rstudio.com) (Figure 1). Because absolute scoring scales require broadly accepted standards (53) and these standards are not evident in the context of emerging infectious disease, for which many characteristic remain unclear or unknown, the WHO tool makes use of a relative scale that compares values between diseases rather than against absolute values (i.e., the impact of scoring diseases A and B at 3 and 5 is the same as scoring the same diseases at 7 and 9).

The data collected were processed by an in-house program implemented in R Studio. A custom analytic hierarchy process (AHP) implementation was used to calculate disease scores for each subcriterion (Technical Appendix 5). This process included normalization and weighting procedures. Comparison matrices were built from data provided by each expert and then averaged by using the geometric average (54). Disease scores for each subcriterion were computed, and an overall multicriteria score for each disease was ultimately computed by using the disease scores and criteria weights. Following best practices, the criteria definition and weighting steps were separated from the disease scoring (5,7,14,36,38). The criteria were defined in 2015 by a group of experts (4) and were then reviewed, validated, and weighted by another group (49).

The 2017 annual review resulted in a list of diseases that pose a risk for a public health emergency and for which an urgent need for R&D exists (50). The earlier ECDC review highlighted numerous general weaknesses among published prioritization processes (5). It identified shortcomings in WHO approaches toward disease prioritization before the R&D Blueprint, including insufficient detail in reporting; a lack of transparency, in particular as to how the prioritization criteria were developed; a need for greater consideration on sources of bias; a better discussion of implementation challenges; methodologic anomalies, such as the use of only a single round of the Delphi technique; and a lack of external review of the methodology and subsequent publications.

The methodology developed by WHO for the R&D Blueprint explicitly addresses these shortcomings (e.g., mitigation of numerous sources of bias). Past methodologic anomalies have also been addressed (e.g., a 2-step semiquantitative Delphi technique is now being used). The reporting process has been strengthened; the methodology has been published in full (3), as has a detailed report of its use during the 2017 annual review (50).

The new approach is also much more transparent, with all publications being more detailed and openly available. These publications explain the reasoning behind why certain diseases ultimately were (or were not) included on the list. The process by which the prioritization criteria were developed (Technical Appendix 2) is also well documented in meeting reports (4,49).

Shortcomings in the review process have been addressed, in part, by separate committees to develop and implement the methodology because these committees effectively review each other’s work. The methodology itself was validated through a dedicated expert consultation, improving the review procedures further. Finally, publication of this article further expands opportunities to review the approach and its implementation.

Several challenges to implementation exist. Other prioritization studies have invested extensive resources into identifying potentially relevant diseases. For example, Cox et al. conducted a bibliometric analysis of >3,000 infectious organisms in North America to identify the 651 pathogens relevant to their study (43). In Belgium, Cardoen et al. complimented a literature review with expert consultations (36). In the Netherlands, Havelaar et al. went a step further, supplementing their literature review with consultations with international, regional, and national experts to identify the relevant subset of pathogens (26).

At present, the community proposing additional diseases to be considered in an annual review of the R&D Blueprint is limited. To address this issue in future prioritization exercises and to better reflect regional factors in the long list of diseases, the WHO regional offices will be more actively involved in the inclusion of a wider range of experts.

The next methodology review should look at the reproducibility of these tools. In the interim, further improvement of the MCDA model might include reviewing the pertinence of the MCMs criterion given that it had little effect on the multicriteria scores of the diseases, reweighting the criteria, drawing on a wider community of relevant expertise and a larger sample size, and reviewing and simplifying the specific wording of the subcriteria.

The R&D Blueprint methodology was developed to mitigate numerous sources of bias, including flaws in study design, selection bias, interviewer bias, chronology bias, and recall bias (3). These efforts were largely successful; however, further work might be necessary to mitigate selection and recall bias.

The selection of experts to participate in the MCDA is important for mitigating selection biases. WHO’s policies on geographic and gender representation go some way to address selection bias. Considerable resources were also expended to create a committee with the diverse range of expertise required, with experts from microbiology and virology, clinical management of severe infections, epidemiology and outbreak investigation and response, public health policy, animal health, mathematical modeling of disease, environmental and social science, nongovernmental organizations, and the security sector. This diversity is consistent with and exceeds the range of participants found in other studies, allowing for some variation based on their specific purposes (8,9,11,13,26,30). Ensuring that future reviews also have a sufficient range of expertise will be important.

The number of experts participating in the annual review meeting also deserves careful consideration. Larger groups increase the likelihood of reproducibility and decrease the risk for certain biases (57). Smaller groups can simplify the consensus-building process (15,58). Group size can also impact group dynamics. Too large a group can make face-to-face consultations impractical, complicating efforts to review and discuss the results, correct eventual inconsistencies, reach consensus, and avoid misunderstanding (8). Although exploring ways that a greater number of experts might be involved with developing an initial long list of diseases to be considered might be useful, a more limited group will probably need to continue to analyze the short list in the years to come.

Additional efforts are also needed to address recall bias. Discussions during the 2017 review highlighted that the diseases that enjoyed the greatest levels of support for inclusion in the revised priority list had all caused recent major outbreaks. An annual “landscape review” (in which each disease on the long list of proposed diseases is independently reviewed, considering factors such as the current knowledge regarding prioritization criteria, risk for emergence, and availability of countermeasures, regardless of recent events) should contribute to avoiding a disproportionate emphasis based on recent events. In the short term, participants in the next annual review should be briefed on, and discuss the impact of, recall bias before undertaking the MCDA scoring exercise. In the longer term, options for weighting against recent public health emergencies might be explored, perhaps through the development of a calibration curve, which has been used to mitigate recall bias in other types of processes (59).

This methodology is expert-driven, and despite all efforts to minimize biases related to their efforts, biases still occur. To address this problem, WHO should 1) change the composition of the prioritization committees yearly and expand the geographic range of the experts involved and 2) review the methodology separately with different experts.

The similarity between the WHO list of prioritized diseases and those found in other studies suggests a degree of consistency with previous findings (8–11,13,14,35,41). The results of the sensitivity analysis demonstrate that the R&D Blueprint ranking is robust, corresponding with earlier observations that the analytic hierarchy process is not sensitive to minor changes in criteria weights (55). Even when major changes on the weight of criteria were applied, the final ranking remained largely stable. Throughout all the scenarios used in this sensitivity analysis, the same 3 groupings of diseases remained consistent. In some scenarios, the ranking of diseases within the group changed, but this observation is consistent with the findings of other prioritization exercises (9). Being able to produce a similar 3-tiered group ranking using another model, the SMART Vaccines prioritization tool, also suggests that the approach employed for the R&D Blueprint is producing valid results.

However, the impact of the MCMs criterion needs further consideration. The sensitivity analysis showed that the contribution of this criterion to the final ranking is limited despite its high weight. This observation is probably explained by the objectives of the R&D Blueprint itself, which focuses on diseases for which few or no MCMs exist, meaning that all the diseases considered score equally in this regard. Ensuring that sufficient attention is paid to this issue when selecting diseases for inclusion on the long-list will be useful as a screening process. Ensuring that distinct R&D gaps are a prerequisite for inclusion could result in this criterion being dropped from the MCDA.

In conclusion, the R&D Blueprint fills a considerable gap in public health preparedness by supporting R&D on highly infectious diseases for which few or no countermeasures exist. To translate this objective into effective action, WHO had to determine the diseases that most urgently required the commencement of work. For each of these priority diseases, WHO is developing roadmaps; target product profiles for vaccines, therapeutics, and diagnostics (60); and generic protocols for vaccine and therapeutic clinical trials. The R&D Blueprint is also enabling cross-cutting support activities, such as data and sample sharing norms, regulatory preparedness aspects, and overall research coordination (61). Aware of the shortcomings of past efforts to develop similar lists, WHO explored lessons learned and best practices for developing a new approach. The challenge was in balancing competing needs for a standardized, robust methodology that can be repeated on a regular basis, with a reliance on expert opinion. Because this methodology and its supporting tools will be subjected to a full review within 2 years, WHO hopes that the lessons learned through the R&D Blueprint’s repeated use, including those we have identified, will be used to improve it further.