Figure 2

Figure 2. Example epidemiologic model output presented to stakeholders as part of decision support tool for coronavirus disease, Utah, USA. Model results compare daily incidence across 3 planning scenarios: no interventions, social...

Utah, like other state, local, and national governments, sought epidemiologic modeling estimates to inform their COVID-19 response. Utah public health decision-makers initially engaged with our group, Infectious Disease Dynamics, at the University of Utah to help prepare for and respond to COVID-19. To address their questions, we adapted a metapopulation Susceptible-Exposed-Infectious-Recovered/Removed modeling process to develop planning scenarios for the state (J.C. Lemaitre et al., unpub. data, https://doi.org/10.1101/2020.06.11.20127894). We projected infections, deaths, and health system needs under multiple nonpharmaceutical interventions (NPIs) being considered by decision-makers (Figure 2). In particular, we compared the effects of comprehensive testing and isolation strategies on the lockdown measures being implemented by other states (e.g., California). Although testing and isolation strategies were not yet feasible in many states because of slow scale-up of testing capacity, Utah was well positioned to take such an approach. As of March 25, 2020, a national diagnostic medicine laboratory located in Salt Lake City had ample resources to rapidly develop and scale up COVID-19 testing capacity.

We compiled the model-based projections and comparison of NPIs and rapidly shared a report on March 23, 2020, with key leadership at the University of Utah Health, the Utah Department of Health (UDOH), ARUP Laboratories (Salt Lake City), the Governor’s Office of Management and Budget, and Intermountain Healthcare, the largest healthcare system in Utah. These stakeholders encompassed the key health decision-makers in the state, including those responsible for ≈60% of the state’s hospital market share.

On March 24, university leadership coordinated a meeting between scientists and policymakers to discuss this initial report. The goal of the meeting was to review model projections, compare the different NPI scenario estimates, and discuss the best paths forward for the state. The resulting consensus was that the state should strive to rapidly achieve levels of per-capita testing of symptomatic persons similar to those seen in South Korea, a goal that was achieved in Utah by March 25, 2020. After this meeting, we maintained open lines of communication with health experts and policymakers, soliciting insight into new operational questions (further discussed in phase 3) and distributing weekly scenario-based projections of probable outcomes under different NPIs over the course of the local outbreak.

Figure 3

Figure 3. Example of a consensus model figure from a decision support tool for coronavirus disease, Utah, USA. Model results compare the number of new reported infections (daily) across the 4 modeling...

The University of Utah model was not the only model used to estimate COVID-19 impact in Utah. In addition to national-level models that included projections for Utah (e.g., projections described in University of Washington Institute for Health Metrics and Evaluation [IHME] COVID-19 Health Service Utilization Forecasting Team et al., unpub. data, https://doi.org/10.1101/2020.04.21.20074732), 3 other groups within the state were developing models of COVID-19 to inform policy. Intermountain extended an existing Susceptible-Infected-Recovered (SIR) model to project expected burden on their healthcare facilities statewide, later switching to a timeseries model for short-term forecasting. UDOH used an SIR model, and another group constructed an operational model of COVID-19 that projected forward on the basis of current trends, thereby implicitly projecting the effect of current NPIs at the state level (group 1 in Figure 3), later moving to a timeseries model for short-term forecasting. The different modeling approaches, which often yielded qualitatively different results (Figure 3), were creating uncertainty about the relative strengths and weaknesses of policy options.

To improve consistency in model outputs and communication of results across the 3 modeling groups, we arranged weekly consensus modeling meetings starting on April 8, 2020, that included representatives from all groups and other stakeholders (e.g., UDOH). Those meetings covered evidence-based model parameters, key modeling scenarios (e.g., determining which NPIs to model), data quality, and appropriate interpretation of high-profile models from outside the state. At these meetings, participants learned that the University of Utah was using a Susceptible-Exposed-Infectious-Recovered/Removed model (later changing to a timeseries model for short-term forecasting) with a latent period of 5 days and an average duration of infection of 6 days, whereas Intermountain was using an SIR model with an average duration of infection of 6 days. Likewise, the University of Utah group assumed that 10% of all infections were in hospitalized case-patients and the duration of hospitalization was on average 11.5 days, whereas Intermountain assumed that 2.5% of infections were in hospitalized case-patients and the duration of hospitalization was on average 7 days. Further, the University of Utah assumed that 15% of hospitalized patients required a stay in the intensive-care unit (ICU), whereas Intermountain assumed that 38% of hospitalized patients required an ICU stay. The consensus modeling group also served as a forum for informal peer review of models from each group. The consensus modeling meetings produced weekly joint reports reflecting the collective research, modeling, and operational efforts of the group, standardizing the outputs (Figure 3) to improve communication. Central to these reports was presenting results from all 3 groups in a format that could enable comparisons, guide public health decision-makers on the strengths and limitations of each model type, and indicate which models were more appropriate for informing certain decisions, such as models that aimed to forecast weekly incidence compared with those aiming to provide big-picture epidemiologic dynamics. To improve communication, these reports began presenting a consensus model, which was calculated as the average of each of the individual group models over the forecast period.

As the epidemic evolved, new operational questions required new approaches. To address these new questions, we contacted collaborators at the University of Utah to develop new decision support tools that expanded the modeling process. In particular, assessing the efficacy of key interventions in a local context became paramount. Doing so required an increased focus on ensuring the model’s assumptions matched the current epidemic situation.

Figure 4

Figure 4. Sample model outputs from additional model components for a decision support tool for coronavirus disease, Utah, USA. Solid lines indicate the average daily occupancy, and shaded areas represent 95% CIs....

To characterize the effectiveness of the NPIs that were implemented in Utah in March 2020, we estimated the time varying local reproduction number, R_t (the real-time average number of secondary infections from a single infected person), with assistance from the Study Design and Biostatistics Center at the University of Utah (Y. Zhang et al., unpub. data, https://doi.org/10.1101/2020.05.08.20095703). Estimates of R_t became a weekly input into the transmission model, and these projections served as a baseline for comparing current and possible interventions (Figure 4, panel A). As the epidemic progressed, local outbreaks sparked concerns of substantial spatial heterogeneity in the impact of interventions across the state. Hence, we began estimating R_t at the county level and capturing this heterogeneity in our wider modeling efforts, as well as including these estimates directly in the report beginning April 13, 2020.

As COVID-19 patients were discharged, public health officials learned that the pandemic would have downstream effects on post–acute-care facilities. These case-patients often require further supportive care after hospitalization; however, they might still be infectious and pose a risk to other long-term care facility residents and staff (10). One week after we shared model outputs with state decision-makers, the state opened a dedicated long-term care facility to accommodate COVID-19 patients. To help calibrate the appropriate capacity of the center and anticipate the need for expansion, we collaborated with a team of hospitalists (general internists who care for hospitalized patients) to extend the process with a module aimed at projecting post–acute-care flows. This module explicitly models the discharge of case-patients directly to home, to home healthcare, to skilled nursing facilities, or to hospice (Figure 4, panel B), and was first included in reports on May 18, 2020 (M. Maloney et al., unpub. data, https://doi.org/10.1101/2020.06.12.20129551).

Although we have devised a process for responding to the ongoing pandemic, the situation continues to evolve. What appears to be effective now might not continue to be fruitful as the outbreak progresses. Likewise, the questions that have arisen thus far represent just a small sample of the potential hurdles that might be faced during a dynamic situation. For instance, we are already working to integrate a health economic model with the post–acute-care components to help guide the development and expansion of additional patient-care resources after hospitalization. In addition, we are beginning to develop collaborations across states with similar experiences, priorities, and concerns to learn from their experiences and further improve pandemic response. Although some future directions are clear, others will emerge as the pandemic evolves. As subsequent outbreaks occur, the response continues to leverage these developed collaborations to provide the state with evidence-based guidance for pandemic response.

We identify 3 key points from the process so far. The first is establishing processes for bidirectional communication among stakeholders, the second is promoting communication and consensus among modeling teams, and the third is inviting multidisciplinary perspectives to inform modeling.

First, ongoing iterative communication with public health officials, policymakers, and other stakeholders is key for developing an understanding of policymakers’ needs and gaining their trust, thereby creating a bidirectional relationship with effective communication. Through the process of producing and sharing weekly scenario-based projections of outcomes with policymakers and health experts, we demonstrated that we incorporated their feedback into the model, offered new interventions and evaluation criteria to consider, and provided support in interpreting the projections. Regular, open communication between stakeholders and modelers also fostered an environment that facilitated conversation between modeling groups and spurred new modeling developments.

The second key point is that debate and discussion of results between modeling groups increased confidence in model results and overall interpretability by policymakers. Before Utah developed its own models, several high-profile, out-of-state models produced unrealistic projections because they failed to account for the local context. For instance, the IHME model predicted hospital capacity would be exceeded in early April, much earlier than was observed, probably a result of drawing parallels with other COVID-19 epidemics based on little evidence and failing to incorporate important contextual details (IHME COVID-19 Health Service Utilization Forecasting Team et al.). The guidance of local models produced a more measured approach to outbreak control (i.e., a rapid scale-up of state testing and isolation), compared with a strict lockdown, which would have been justified to prevent the dire hospital overflow predicted by other models. The interagency collaboration developed through the consensus group helped to draw on diverse perspectives, account for local context, and boost confidence in model projections statewide. Importantly, comparing multiple models helped refute the false narrative that differing models are necessarily in competition. This comparison helped to highlight to both the consumers of the results and the individual modeling teams that each model is a tool optimized for addressing a particular type of policy question by making certain assumptions.

Finally, modeling approaches need to be adaptable and multidisciplinary to address changing policy questions. By using a solution-oriented modular approach, we were able to adjust and expand the initial epidemiologic model to assess how using an NPI affected the number of cases, the number of hospital or ICU beds needed in the short term, and the number of skilled nursing facility beds needed on a longer time scale, as well as, ultimately, the effectiveness of the NPIs used. An additional benefit of incorporating multiple modeling components was the differing perspectives in evaluating model assumptions and interpreting outputs gained by collaborating with experts from a range of disciplines. This collaboration between epidemiologists, health economists, biostatisticians, and hospitalists yielded perspectives beyond any single discipline and enabled groups to focus on modeling within their areas of expertise. Each model component was developed as a separate module, but results were shared regularly to solicit feedback, determine how they would inform the other modules, and formulate a consistent message for stakeholders.

In conclusion, the framework we have described can be applied in other settings to address additional public health challenges. This approach is best used at the level that decisions are being made and policies put into place. Each jurisdiction, whether at the city, county, state, or regional level, has its own particular conditions that affect disease transmission and number of cases (e.g., population density and demographics), and which intervention and treatment options are feasible (e.g., local laboratory capacity to scale up testing). As a result, modeling approaches for the same public health threat are bound to vary. An interdisciplinary modeling hub with university-level support for these kinds of cross-cutting collaborations, such as the one we created, would enable the kind of inclusive, rigorous exchange that can yield valid models and estimates that multiple modeling groups can support. By enabling sharing of modeling approaches and sustaining dialogue focused on policymakers’ questions, the forum would help modelers propose relevant and operationalizable scenarios that will probably resonate with policymakers and result in greater uptake. Another strategy would be to apply this multidisciplinary approach at the national level; however, a continuous dialogue between modelers, experts on the varied local conditions, and local politicians would be integral for the success of a national-level response.