Enhancing Workforce Capacity to Improve Vaccination Data Quality, Uganda

In Uganda, vaccine dose administration data are often not available or are of insufficient quality to optimally plan, monitor, and evaluate program performance. A collaboration of partners aimed to address these key issues by deploying data improvement teams (DITs) to improve data collection, management, analysis, and use in district health offices and health facilities. During November 2014–September 2016, DITs visited all districts and 89% of health facilities in Uganda. DITs identified gaps in awareness and processes, assessed accuracy of data, and provided on-the-job training to strengthen systems and improve healthcare workers’ knowledge and skills in data quality. Inaccurate data were observed primarily at the health facility level. Improvements in data management and collection practices were observed, although routine follow-up and accountability will be needed to sustain change. The DIT strategy offers a useful approach to enhancing the quality of health data.

O ptimal immunization coverage against vaccine-preventable diseases (VPDs) is essential for achieving and maintaining global health security. Obtaining such coverage relies on high-quality immunization data, which are a prerequisite for good decision making; effective and efficient public health action, monitoring, and evaluation; and improved population immunity against VPDs (1)(2)(3). Enhanced demand for vaccination data and scrutiny of their quality are evident in strategic guidance documents for the Global Polio Eradication Initiative (GPEI) (4), the Global Vaccine Action Plan (5), and the recently introduced data quality requirements for financial support from Gavi, the Vaccine Alliance (6). Availability and quality of vaccination data are often inadequate to inform policy, effective management, and monitoring of vaccination programs (3,7,8).
In 2013, Uganda conducted a national data quality selfassessment (DQS) (9) (Ministry of Health, Uganda, unpub. data) and found that the quality of administrative vaccination data was suboptimal, particularly at the subnational level, which was likely contributing to inflation of administrative coverage data (10). Reasons for poor data quality included inaccurate vaccine dose administration data generated at the health facility, deficiencies in healthcare worker knowledge and skills, scarcity of standard recording and reporting tools, and inadequate implementation of recommended practices for data management collection, analysis, and use. Many of these issues had been previously identified in Uganda and elsewhere (8,(10)(11)(12). To guide implementation of recommendations from the DQS, the technical working group for the Ugandan National Expanded Program on Immunization (UNEPI) developed the National Data Quality Improvement Plan. This plan laid out how, and at what level, the recommendations would be addressed, recognizing limited published evidence regarding effectiveness of specific approaches to strengthen immunization data quality (3,12,13). Given the importance of an effective workforce, a central component of the Data Quality Improvement Plan was to enhance the capacity of existing healthcare workers to manage, analyze, and use vaccination data. The chosen approach was guided by growing evidence supporting on-the-job training of healthcare workers that includes feedback and follow-up (14), which had previously been used successfully in Uganda (15,16). This article describes the initial implementation (November 2014-September 2016) and outcomes of Uganda's national strategy to improve administrative vaccination data quality, defined by the dimensions of management; collection; data produced (accuracy, timeliness, completeness); analysis; and use (17).

Preparation for the Data Improvement Team Strategy
The data improvement team (DIT) strategy was developed and managed by a national DIT strategy management group, which included UNEPI, the Resource Center (the responsible entity for managing health information) of the Uganda Ministry of Health, World Health Organization (WHO) Uganda, the US Centers for Disease Control and Prevention (CDC), the African Field Epidemiology Network (AFENET), UNICEF, and Gavi. Implementation was funded jointly by Gavi Health Systems Strengthening Grant 1, WHO, UNICEF, and CDC and led by a national coordinator from AFENET, with technical assistance from CDC.
The strategy aimed to strengthen the immunization information system and quality of the resultant data at the district and health facility levels through practical classroom training, deployments involving rapid data quality and organizational assessments, and on-the-job training ( Figure 1). The number of DIT members required for each district was determined on the basis of ability to reach all health facilities that provided immunization services in that district (range 6-117) within 5 to 6 working days, spending 2 to 3 hours at each. A district-level DIT included an average of 4 district staff members (with additional members in high-population areas) and 1 Makerere University School of Public Health (MakSPH) student. Districts were asked to identify staff to form a DIT, which included the district biostatistician, district Expanded Programme on Immunization (EPI) and surveillance focal persons, and a health records assistant. MakSPH staff and the national DIT coordinator led recruitment of students.
The DIT strategy was designed to be implemented in a phased approach by region ( Figure 2); several district-level DITs were trained together, then deployed in their respective districts. All official government districts in Uganda as of November 2014 were divided into 17 DIT operational regions to ensure that the number of attendees at regional training was logistically manageable and there was close geographic proximity between districts in each region.

Training
Before implementation, a 5-day orientation to the strategy and Uganda's immunization information systems was provided to national staff, who self-selected to support delivery of the regional-level training and to conduct supportive supervision of DIT activities. The 3-day regional training aimed to build selected DIT members' knowledge and skills in data management and quality, which were applicable during the DIT deployment and their regular duties thereafter ( Figure 1).

Deployment
In the week after each regional training, DIT members were deployed to their home districts for 5 to 6 days to work at the district office and visit health facilities (Figure 1). Working in pairs, DIT members identified problems, proposed solutions, developed recommendations, and enhanced staff capacity through on-the-job training on locally identified problems (e.g., how to create an immunization monitoring chart) ( Figure 2). DITs initially prioritized health facilities with outlying (high or low) coverage for the third dose of the diphtheria/tetanus/pertussis/Haemophilus influenzae type B/hepatitis B vaccine (Penta3), negative dropout rates, or inadequate completeness or timeliness of Health Management Information System (HMIS) monthly reports (18). Staff from the Ministry of Health and national EPI program partner organizations provided supportive supervision to DIT activities S86 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 23, Supplement to December 2017 Overview of processes and key activities of the Uganda data improvement team (DIT) strategy to improve vaccination data quality. At the center are elements that are ongoing throughout implementation of the 4 main activities: financial, technical, and logistical collaboration between Expanded Program on Immunization partners, coordination provided by a DIT strategy management group and the DIT national coordinator, and routine monitoring and evaluation. Preparation includes discussing and developing budget, designing the approach to implementation and materials for training and monitoring and evaluation, training supervisors, grouping districts into regions, and identifying DIT members. For training, grouped by region, DIT members from several districts attend a 3-day training led by staff from the Ministry of Health Expanded Program on Immunization and the DIT strategy national coordinator. This training included a combination of technical lectures, practical case studies (80% of all sessions), and a practice visit to a health facility (half-day). Deployment core activities include district and health facility organizational assessment and a rapid data quality improvement questionnaire to identify strengths and gaps in resources and systems for immunization data management, collection, analysis, and use. Results inform recommendations developed by the DIT members who provide on-the-job training of staff to strengthen action on recommendations. DIT members debrief leadership (region, district, health facility) on findings and recommendations, and harness support to implement recommendations. Finally, national DIT strategy management groups review activities and results at several time points ( Figure 2); based on evidence from implementation and current national priorities, the strategic and operational approaches are revised, then reimplemented.
in some districts, assisting coordination and implementation of activities, conveying national-level support for the DIT strategy to district leadership, and enhancing their own awareness of ground-level operations.

Monitoring and Evaluation
A participatory and utilization-focused (19,20) approach was taken to routine monitoring and evaluation of processes, outputs, and short-term outcomes. Training was evaluated through a self-administered survey focused on quality of the training experience; a pretest and posttest measured participants' acquisition of knowledge and level of preparedness to implement DIT activities. DITs conducted an organizational assessment at the district and health facility levels to inform their work and to gather baseline information on key indicators (21). Organizational assessments contained a mix of closed and open questions covering dimensions of, and factors affecting, vaccination data quality. Results of organizational assessments were reported to the DIT national coordinator either through a reporting template in Excel (Microsoft, Redmond, WA, USA) (106 districts, 1 Kampala division) or by using an open data kit-based mobile application linked to a cloud-based database (5 districts, 4 Kampala divisions). At the health facility, DITs also used a data quality improvement (DQI) questionnaire to review practices for data management, collection, accuracy, analysis, and use ( Table  1). The primary purpose of this questionnaire was to identify gaps that would inform recommendations and on-thejob training. For purposes of analysis for monitoring and evaluation, DQI questionnaires from health facilities were sampled from 107 of the 116 districts (92%) for which these data were not reported through the mobile application. The sample included all hospitals and every second health facility selected from an alphabetized list, until the sample size reached 50% of all health facilities in the district. In an additional 7 districts, DQI reports from all visited health facilities were entered in the mobile application. Descriptions of the DIT activities, outputs, and recommendations were presented in a written report for each district health management team. Line-listed results from organizational assessments and DQI questionnaires were aggregated nationally and quantitative data were descriptively analyzed in SAS version 9.3 (22) and Tableau version 9.3.1 (23). The sign test was used to assess the statistical significance of the direction of difference between sources of vaccine dose administration data and was performed in R version 1.5.1 (24).
After DITs had been deployed to all districts, a review of DIT implementation was undertaken to gather feedback about the approach and understand extent of action on recommendations through a rapid organizationallevel survey in a sample of districts and health facilities. Four regions were selected from the 17 DIT operational regions; 2 or 3 districts were selected from each region, and within each of these, 4 health facilities were selected, totaling 11 districts and 44 health facilities. If a selected site could not be visited, it was replaced with the next one of the same type on an alphabetized list of health facilities in the district. Selection was purposeful to gain insights across a range of characteristics, including geographic location, implementation of national supervision, Reaching Every District categories (25), and level (type) of health facility (26). Eight data collectors (4 AFENET/CDC staff and 4 MakSPH students) completed a 1-day training, then worked in pairs to visit the selected sites to conduct the survey through group interviews with district and health facility staff. Resultant data were descriptively analyzed in Epi Info software (27).
Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 23, Supplement to December 2017 S87 The proportion of health facilities in a district submitting monthly HMIS reports on time to the district (timeliness) and the proportion of expected reports received by the district (completeness) are routinely calculated in the national electronic HMIS (12,18). In districts for which these data were available for the 3 months and after the DIT visit (n = 104) and for the second 3-month period after the DIT visit (n = 95), timeliness and completeness, by month and district, were extracted from the electronic HMIS. Median timeliness and completeness were calculated per district across each 3-month period, then compared between periods to identify change.

Review and Revision
The national DIT strategy management group held periodic meetings ( Figure 2) to review results from monitoring and evaluation and the budget, as well as to solicit feedback from all stakeholders. These meetings, in conjunction with national priorities, informed any adjustment of DIT activities and implementation plan.

Training and Deployment
During November 2014-September 2016, all 112 districts and 5 divisions of Kampala (total 116 DIT operational districts) in Uganda sent staff to DIT regional training and deployed district-level DITs. Seventeen regional trainings, covering 2-14 districts per training, attended by 451 district and health subdistrict staff and 35 MakSPH students (some attended multiple trainings [range [1][2][3][4][5][6][7][8][9]). In response to participant and stakeholder feedback, the training format was altered to enhance the balance between the practical and didactic sessions (Figure 2 (355/429) of district staff demonstrated improved knowledge on posttest compared with pretest scores, and more participants felt "fully prepared" to conduct DIT activities (14% pretest, 82% posttest). In total, 476 DIT members (including 35 MakSPH students) were deployed and reached 89% of health facilities that provided immunization services (Table 1). Health facilities not visited (n = 413) were predominantly health center IIs (HCIIs; n = 332, 80%), which offer a limited number of services, serve smaller catchment areas, and are often geographically remote. Initially, DITs reviewed paper copies of monthly HMIS reports from health facilities submitted to the district office and compared doses reported for all antigens with those recorded in the electronic HMIS for the 12 months before the DIT visit (Table  1). Time spent on this activity reduced the time available to reach all priority health facilities by an average of 8 hours per district. Because early results showed high congruence between these 2 data sources (Figure 3, panel D), this activity ceased after the midterm review meeting, enabling DITs additional time to conduct organizational and DQI assessments and develop recommendations for improvement (average 1.2 hours per health facility) and to implement on-the-job training (average 1.5 hours per health facility).
Through the organizational assessment, DQI questionnaire, and discussions with staff, DITs identified a combination of external factors, often specific to the site visited, that affected vaccination data collection, management, analysis, and use. Commonly identified challenges included poorly motivated, new, untrained, or absent staff; unavailability of materials for recording and reporting  data; competing priorities on staff time due to integration of services; inadequate supportive supervision for data quality; limited transport, technological, and financial resources; variable understanding and commitment by political or organizational leaders; and competition with other public health initiatives for human, financial, and material resources. In Uganda, doses of vaccines administered are recorded on 4 tools: tally sheet, child register, and monthly report (at health facility) and the electronic HMIS (entered at district level using data from health facilities' monthly report). We found variable congruence between monthly totals of vaccine doses across these 4 sources for any given month (Figure 3). On average, the number of administered doses aggregated on the monthly report was higher than that recorded individually on the tally sheet ( Figure 3, panel C), which was higher than that recorded on the child register ( Figure 3, panel A). This finding suggests that vaccine administration is overreported by the health facility and that use of the child register is low compared with other sources of vaccine dose administration data (Figure 3, panels A, B). We found stronger agreement between the number of doses on the paper HMIS monthly report and those in the electronic HMIS ( Figure 3, panel D), highlighting infrequent transcription error or loss of data from district to national level. There was individual variation in the discordance by health facility, with no clear pattern by district or health facility type. Similar patterns in data congruence were also seen for single-dose measles vaccine offered to older children (data not shown).

Postimplementation Follow-Up
The postimplementation follow-up survey found that DITs had visited all sampled districts (n = 11) and 77% (34/44) of sampled health facilities. Recommendations provided by DITs addressed all dimensions of data quality; however, the extent of implementation varied ( Table 2). Recommendations for each district most frequently related to improving systems for archiving, checking data on monthly HMIS reports, and charting coverage data. At the district level, recommendations relating to data management and collection were more fully implemented than those related to analysis and use ( Table 2). Recommendations for health facilities most commonly focused on improving recording and reporting of data, analysis, and archiving. Recommendations related to management and collection were more completely implemented than those related to analysis. No health facility reported taking action on recommendations to improve data use ( Table 2). Reasons for inaction across all recommendations included insufficient availability of required materials (standard data collection/reporting tools, archiving space); inadequate human resource capacity (new staff, untrained staff, low motivation); and a management structure that limited staff awareness of, and roles in, immunization data collection, management, analysis, and use.
During the follow-up survey, district staff frequently reported that participation in the DIT activities catalyzed improvements in existing, or development of new, systems and processes, such as supportive supervision about vaccination data quality. Health facility staff felt that the visit by the DITs was a catalyst for provision of updated recording and reporting tools and helped them develop systems to S90 Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 23, Supplement to December 2017 enhance completeness and accuracy of data reported on the monthly HMIS report (Table 3). Timeliness and completeness of HMIS monthly reporting (from health facility to district) averages >90% nationally (28). This high performance limits the opportunity for and measurement of change; however, there was some improvement. Comparing 3 months before and after DIT implementation, 15% (15/104) of districts showed improvement in completeness, 6% (10/104) decreased completeness, and the remainder no change. From the initial 3 months to the second 3 months post-DIT implementation, completeness improved in 25% (24/95) of districts, decreased in 10% (9/95), and showed no change in the remainder. More districts showed improvement in timeliness of monthly HMIS reporting. Comparing 3 months before DIT implementation to 3 months after, 38% (40/104) improved, 20% (21/104) decreased, and the remainder showed no change in timeliness. From the first to second 3-month periods after implementation, 27% (26/95) of districts showed improvement, 50% (47/95) decreased, and the remainder showed no change.

Discussion
EPI partners in Uganda took a collaborative approach to developing, funding, and implementing a strategy to address recommendations from Uganda's most recent DQS. Over 23 months, 351 district staff and 35 MakPSH students were trained and 479 DIT members were deployed, in phases, to all districts and 89% of health facilities that provide immunization services in Uganda. Rapid assessments of organizational-level immunization information systems and accuracy of resultant data identified gaps in skills and systems for data management, collection, analysis, and use. Assessments indicated that the child register was underused, and the tally sheet was used as the primary data recording tool, with greater variation in the difference between these primary data sources than for data aggregated at the district and national levels. Timeliness and completeness of HMIS monthly reports from health facilities was high at baseline; although some districts showed improvement, there was volatility in these changes. Recommendations for improvement and changes made by district and health facilities related predominantly to strengthening systems and processes, with those related to management and collection more completely implemented than those related to analysis and use.
DITs identified that poor data quality stemmed largely from inaccurate and incomplete recording and reporting of vaccine dose administration data at the health facility and poorly implemented processes for data management, collection, analysis, and use. These problems likely contributed to overreporting of administrative data, as identified in the 2013 Uganda DQS (10). If data are improperly recorded at, or inaccurately reported from, the health facility to the district level, these data will remain inaccurate in the national HMIS (18). Although data are prone to errors such as incorrect entry, incompleteness, or late reporting, accurate recording and reporting of vaccine doses administered from the initial point at which they are generated is critical to improving the quality and utility of data at all levels of the health system (29). The relationship between data quality and use could be considered cyclical, in that improving accuracy could improve confidence in the data, which would help drive demand and use, further driving data quality. At a service delivery level, if data are not used to monitor performance, opportunities can be missed to identify issues as they arise, such as problems with dropouts, changes in target population, or underserved areas, all of which can lead to underimmunized children and can leave the population vulnerable to outbreaks of epidemicprone VPDs, which threatens global health security (30).
Improving data accuracy in a situation of overreporting may result in lower immunization coverage estimates (7). Despite implying poorer program performance, increased accuracy would enhance the utility of the data for informing immunization program implementation, including identification of underimmunized or nonimmunized populations that may have been masked by overreporting.
Emerging Infectious Diseases • www.cdc.gov/eid • Vol. 23, Supplement to December 2017 S91 Table 3. Extent of self-reported changes catalyzed by the DIT visit to improve vaccination data quality in select districts and health facilities in Uganda* Underrecording of individual-level vaccination status in the child register inhibits the ability of healthcare workers to identify and follow up with inadequately immunized children, both routinely to maximize coverage and during VPD outbreaks. Underrecording also reduces the utility of the child register as a secondary data source to verify caretakers' recall when home-based vaccination records are not available (31). Home-based records enable health facility staff to routinely verify individual vaccination status and are critical to the success of periodic independent coverage surveys, which are valuable to verify administrative vaccination data. However, discordance between sources of data on vaccination coverage and inherent limitations in many sources of vaccination data make it difficult to determine true immunization coverage. Some components of the DIT strategy are not typical of other approaches to national data quality improvement initiatives and could be applicable to other countries and other health data. First, the strategy was facilitated through a hybrid funding commitment across multiple organizations, which allowed it to be implemented nationally. Second, the combination of site-specific problem identification followed by immediate, on-the-job training was found to be a useful approach to strengthening healthcare workers' awareness, knowledge, and skills. A similar package of interventions has been seen to improve the quality of supportive supervision for immunization in Georgia (13). Systematic literature reviews highlight the effectiveness of multifaceted approaches, which include audit, feedback, and supportive supervision, in building health workforce capacity (14,32). The capacity to understand the gaps and challenges faced and to tailor improvement strategies accordingly appears fundamental to improving immunization coverage (33). Third, involvement of existing national and district staff helped build sustainability. Finally, MakSPH students, many of whom were redeployed several times, developed their own knowledge and skills, which they felt enhanced their future job prospects. They also brought an external eye that enhanced problem detection, accountability, and external motivation of DITs and health facility staff.
There are limitations to individual methods used for monitoring and evaluation of the DIT strategy, although in combination they facilitated a better understanding about implementation and short-term change (34). DIT members and data collectors were trained in the use of data collection instruments, standard question prompts were included, and data validation was built into the mobile application. Systematic sampling of DQI tools for analysis reduced some systematic error and improved internal validity of these data. The magnitude of difference between sources of vaccination data was influenced by variation in month of assessment and number of doses reported, which was, in turn, a function of health facility type. Administrative data on timeliness and completeness of reported vaccination data are likely limited in specificity and internal validity. Feasibility influenced purposive selection of sites for the postimplementation follow-up, which was also open to researcher bias, although use of selection criteria helped reduce this (35). Different data collection methods were used for routine monitoring and postimplementation follow-up, which did not allow for extensive quantitative comparison between resultant data. Unless directly attributed through individual report, observed changes could not be credited solely to the DIT strategy.
Implementation of the first phase of the DIT strategy catalyzed stronger administrative vaccination data in Uganda. Informed by these experiences and results, a second round of DIT visits to all districts, targeting all health facilities, is being implemented. Planned modifications include follow-up to further determine extent of implementation of recommendations at all sites and degree of shortterm change, as well as regular regional-level meetings of districts to improve accountability and drive action on recommendations. Assessment of vaccination data congruence will continue to focus on the health facility, although assessment of this across the immunization information system should be undertaken periodically to rule out any systematic data entry error or loss of data. The DIT strategy and observed changes have the potential to benefit data from other health initiatives, particularly those reported through the HMIS. Other countries looking to address vaccination data quality issues should consider a similar approach, using existing staff, on-the-job training, mechanisms for routine follow-up, and collaborative resource mobilization. Efforts should focus on identifying site-specific issues and building local workforce knowledge, skills, and awareness, as well as strengthening systems, to enhance availability, quality, and use of vaccination data. and Prevention, Atlanta, Georgia, USA. Her research interests include implementation research for routine immunization programs, surveillance of vaccine-preventable diseases and those of global health significance, as well as program monitoring and evaluation.