We used the Electronic Medical Record Support for Public Health (ESP, https://www.esphealth.org) public health surveillance platform to develop the RAVIOLI algorithm. ESP is open-source software that uses automated daily extracts of EHR data to identify and report conditions of public health interest to health departments (13–17). ESP maps raw data to common terms and then applies algorithms to identify conditions using diagnosis codes, prescriptions, laboratory tests, and vital signs. In Massachusetts, ESP is used for automated reporting of infectious disease cases to the Massachusetts Department of Public Health, aggregate reporting of chronic diseases, and continuum-of-care assessments (18–21).

Three multisite clinical practice groups that use ESP for infectious disease reporting, Atrius Health, Cambridge Health Alliance, and Boston Medical Center, contributed data for our project. Atrius Health (https://www.atriushealth.org) is an ambulatory care group with >30 locations in eastern Massachusetts that provides clinical services for a population of ≈700,000. Cambridge Health Alliance (https://www.challiance.org) is a safety-net system that provides ambulatory and inpatient care to >140,000 patients in communities north of Boston. Boston Medical Center (https://www.bmc.org) is a 514-bed academic medical center and safety-net hospital that provides ambulatory and inpatient care to ≈220,000 persons. We combined data from those 3 sites for this analysis.

We sought to develop an evidence-based set of diagnosis codes to identify respiratory virus–like illnesses and assess whether a subset of those codes might be predictive of specific pathogens. To identify codes associated with respiratory viral illness syndrome, we identified all patients tested for respiratory viruses (Table 1) during October 3, 2015–July 30, 2022. Among patients who tested positive for >1 virus, we identified all International Classification of Diseases, 10th Revision (ICD-10), diagnosis codes recorded within 2 days before or after the specimen collection date. For patients without a recorded specimen collection date, we used the test order date; if that was unavailable, we used the result date. We manually removed ICD-10 codes unrelated to respiratory viral illness (e.g., trauma, cancer, chronic disease management). The list of >7,000 excluded codes is available upon request from the authors.

We calculated the positive predictive value (PPV) for each ICD-10 code associated with positive respiratory virus test results. We also calculated the PPV for measured temperature >100°F within 2 days before or after a positive respiratory virus test. We calculated the PPV for each ICD-10 code and fever as the number of encounters with the diagnosis code within 2 days of a positive test divided by the total number of times the diagnosis code occurred across 
all clinical encounters during the study period. We defined a clinical encounter as a patient receiving a relevant diagnosis code, immunization, vital sign measure, laboratory test, or prescription.

We included in the final algorithm diagnosis codes with a PPV ≥10% for any respiratory virus (all viruses combined) or for a specific individual respiratory virus. We also included encounters with positive respiratory virus tests in the total count of respiratory virus encounters as well as in virus-specific categories of RAVIOLI. We counted each viral encounter only 1 time if the patient had both a positive respiratory virus assay result and >1 suggestive diagnosis code. We classified measured fever alone and diagnosis codes with a PPV of ≥10% for any positive respiratory virus test but <10% for any specific respiratory virus in a category referred to as nonspecific for respiratory viral illness syndrome. In summary, we categorized positive cases within RAVIOLI as virus-specific (e.g., influenza, adenovirus), based on a positive test or a diagnosis code with a PPV ≥10% for the specific virus, or nonspecific, based on fever or a diagnosis code with a PPV ≥10% for any positive test of interest.

To better understand the underlying data included in the final RAVIOLI algorithm, we examined the proportion of patients in each virus-specific category of the algorithm with a positive laboratory test and the proportion of patients in the nonspecific category with a fever. We generated weekly counts during October 3, 2015–January 13, 2024, for clinical encounters with patients meeting the RAVIOLI algorithm, overall and stratified by the probable etiology when possible. For comparison, we also identified the proportion of patients that met the definition of ILI: fever and a diagnosis code for any influenza-like symptom or diagnosis; fever was identified by either a measured fever >100°F or diagnosis code for fever (Appendix Table 1).

Figure 1

Figure 1. Numbers of patients with a clinical encounter for respiratory virus–like illness and the percentages that met the requirements for influenza-like illness versus those of the RAVIOLI algorithm for monitoring respiratory...

Forty-two diagnosis codes (Table 2) and measured fever (>100°F) had a PPV ≥10% for either any positive respiratory virus test (nonspecific) or >1 virus-specific positive test; those diagnosis codes and fever are included in the RAVIOLI algorithm. We recorded weekly counts of patients with clinical encounters and calculated the proportion that met the definition for RAVIOLI overall (diagnosis code, fever, or positive respiratory virus test) and, for comparison, the proportion that met the ILI criteria (Figure 1). The percentage of encounters that met the RAVIOLI algorithm showed clear seasonal trends of annual winter spikes during 2015–2019 followed by periodic increases during spring 2020–early 2024, corresponding to emergence or surges of SARS-CoV-2, RSV, and influenza in Massachusetts. RAVIOLI was identified in a much larger proportion of encounters than ILI after March 2020 and, at times (e.g., fall 2021, August–November 2023), ILI did not detect an increase in respiratory virus illness while RAVIOLI did.

Figure 2

Figure 2. Numbers of patients that met the requirements for the RAVIOLI algorithm for monitoring respiratory virus–like illness, by pathogen category and week, Massachusetts, USA, October 2015–January 2024. A) October 2015–January 2024;...

Figure 3

Figure 3. Percentage of patients meeting the RAVIOLI algorithm for monitoring respiratory virus–like illness, by age group, Massachusetts, USA, October 2015–January 2024.

We estimated weekly counts of patients with clinical encounters meeting the RAVIOLI algorithm stratified by encounters with virus-specific or nonspecific encounters without a classified virus. We calculated those data for the full study period, October 2015–January 2024 (Figure 2, panel A), and for January 2020–January 2024 (Figure 2, panel B). Before March 2020, most RAVIOLI encounters came from the influenza or nonspecific categories. SARS-CoV-2 subsequently dominated until fall 2021, when the nonspecific category reemerged, along with influenza and RSV. When we examined trends by patient age groups, the highest proportion of encounters that met the RAVIOLI algorithm were among children 0–4 years of age, followed by young persons 5–24 years of age (Figure 3).

Data from January 2023–January 2024 show the proportions of patients in the COVID-19, influenza, and RSV categories with a positive laboratory test versus diagnosis code, as well as the proportion in the nonspecific category with fever (Appendix Table 2). The proportion with a positive test varied by virus and time; patients in the COVID-19 category were least likely and those in the RSV category most likely to have a positive laboratory test. Among patients in the nonspecific category, one third or fewer had evidence of fever, and most were identified by a diagnosis code. We also determined the proportion of RAVIOLI patients identified on the basis of >1 positive laboratory test, diagnosis code, or fever during January 2021–January 2024 (Appendix Figure 1); RAVIOLI patients can meet >1 criterion (e.g., have both a positive laboratory test and a diagnosis code). Diagnosis codes were the most common element contributing to identification in most weeks, followed by positive laboratory tests and fever.

Respiratory viruses continue to impose a high burden on patients, healthcare providers, and society, and multiple pathogens, including SARS-CoV-2, influenza, RSV, and others, contribute to the burden of respiratory illnesses. Both healthcare providers and public health agencies therefore have an interest in having access to timely and granular data on trends in respiratory viral illnesses and contributing pathogens. We developed an EHR-based algorithm for integrated surveillance of respiratory virus illness syndromes and associated pathogens using historical data to identify diagnosis codes and other characteristics of healthcare visits most predictive of confirmed respiratory viral illnesses. The RAVIOLI algorithm comprises positive laboratory tests, evidence-based diagnosis codes, and measured fever.

We have implemented RAVIOLI surveillance within the ESP automated public health surveillance platform to provide the Massachusetts Department of Public Health and participating practices with weekly reports on RAVIOLI incidence and contributing pathogens. RAVIOLI provides the department and practices with granular insight into evolving trends in respiratory viral illness rates that both retains the best features of traditional syndromic surveillance (capacity to monitor changes in disease incidence in near real time regardless of whether persons get tested) and simultaneously broadens the scope of surveillance to include multiple pathogens, not just influenza and SARS-CoV-2. The data provide insight into the relative proportions of contributing pathogens across multiple clinical facilities using both test results and diagnosis codes to identify organisms.

When implemented well, syndromic surveillance provides a picture of the frequency, intensity, and trends in indicators of infectious and noninfectious conditions at local and extended scales. Integrating available viral pathogen test results, even if only in a subset of the population under surveillance, as we have done with the RAVIOLI algorithm, can add information about what is or is not contributing to observed increases in respiratory viral activity. Although influenza-like illness and COVID-like illness surveillance have been critical components for monitoring influenza and COVID-19 activity, reliance on fever as a required component of syndromic definitions is problematic because fever occurs only in a minority of laboratory-confirmed influenza and SARS-CoV-2 cases (22–24). Syndromic surveillance algorithms that require fever can therefore miss critical trends in the incidence of illnesses (9). The RAVIOLI algorithm, in contrast, does not require fever as a criterion and uses both laboratory test results and an evidence-based set of diagnosis codes to increase both sensitivity and specificity.

Limitations of RAVIOLI surveillance include its development in a single region of the country using data from just 3 practice groups. Generalizability to other practice groups and regions need to be assessed. Changes in testing practices or coding practices over time and between practices might change the future performance of the RAVIOLI algorithm. The algorithm will require periodic revalidation and possibly modification. Furthermore, the breadth of pathogen capture using the RAVIOLI algorithm depends on the range and frequency of respiratory viral testing by clinicians; greater use of multiplex testing platforms will provide more granular and robust results. RAVIOLI surveillance is limited to patients who seek care, which likely biases the data toward pathogens associated with more severe disease. The PPV of algorithm components may vary by season; whether and how this affects surveillance should be considered. We used a 10% PPV threshold to select diagnosis codes for inclusion. This threshold was arbitrary, but we found using higher thresholds dramatically reduced the number of eligible diagnosis codes. We also found that the terms associated with diagnosis codes with a PPV of ≥10% were specific in their descriptions and not indicative of broad health conditions. However, the PPV threshold for including diagnosis codes should be considered in future revalidation of the algorithm.

The healthcare site data included in developing the algorithm and whose data are part of the weekly reports came from both ambulatory and inpatient care facilities. We observed variation in which RAVIOLI categories (e.g., influenza, RSV) of the algorithm were detected at each site (data not shown). The limited number of sites makes it difficult to know if apparent differences between ambulatory and inpatient sites resulted from differences in catchment populations, illness severity associated with different viruses, or testing platforms. As the network expands to include a greater number and variety of sites, we plan to examine this question further.

The Massachusetts Department of Public Health has used data from the underlying EHR-based system for infectious disease reporting and surveillance for more than a decade (18–21,25–28). This system has been sustained and enhanced over time to meet MDPH needs. As public health agencies consider what they need for the monitoring of current, emerging, and as-yet unidentified pathogens, we have found that a robust EHR data platform is a critical complement to traditional surveillance data.

In conclusion, we developed an integrated, routine, automated EHR-based system for respiratory virus surveillance in Massachusetts. As experience with this approach expands, the hope is that this system will provide early indications of emerging infection trends and prevailing pathogens that render a fuller picture of respiratory viral activity beyond ILI and COVID-like illnesses. A broader view of circulating pathogens will provide public health agencies and healthcare institutions with more precise information useful for informing testing guidance, optimizing health communications; developing more targeted prevention activities, including vaccination; initiating enhanced infection control measures, such as masking and posting of notices in facilities; and generating other policies optimized to minimize the effect on population health of specific circulating pathogens.