Usefulness of Evaluations

Six interventions were evaluated (Table 2); antimicrobial drug–coated catheters were included in 3 separate analyses (27,29,30). One intervention was compared with current practice for all studies, except those of Shorr et al. (29) and Ritchey et al. (28), who evaluated 3 types of antimicrobial drug–coated catheter and 3 different catheter replacement regimens, respectively. No direct comparisons were made across intervention types, e.g., use of an antiseptic catheter versus introduction of chlorhexidine as a skin preparation, and no evaluations assessed multiple concurrent interventions or bundles. The authors of 6 evaluations (23,24,26,27,29,30) found the intervention to be effective in preventing CR-BSI and cost-saving (Table 3), and the authors of 2 other evaluations (25,28) generated data to calculate incremental cost-effectiveness ratios.

Sensitivity analysis was performed in addition to baseline analysis in 5 evaluations (23,26,27,29,30). This provided decision makers with information on the robustness of baseline results to different parameter estimates or characterized the effect of uncertainty in model parameters on the results (23,27,30). In 3 cases (24,25,28), sensitivity analysis formed the main body of the evaluation, and decision makers faced multiple sets of results arising from different parameter estimates.

Quality of Economic Evaluations

The extent to which the quality criteria were met for the studies varied from 1/8 for checks on the internal consistency to 8/8 for description of strategies/comparators. This assessment is shown in Table 4.

Model Structure

All authors provided a clear description of the intervention and specified the economic perspective used, which in all cases was that of the hospital or healthcare payer rather than a societal perspective. Only Shorr et al. (29) justified their choice of perspective. In 7 evaluations (23,24,26–30), a decision tree was used, with a diagram provided in all but 1 report (26). In another evaluation (25), a regression model was used, and only the formula used for the baseline analysis, not the extension used for sensitivity analysis, was provided. Authors of only 4 evaluations discussed the evidence or expert opinion used to develop the structure of the model (23,27,29,30).

Each evaluation used a different representation of the disease pathway in terms of the timing and nature of the relevant clinical events. For example, 1 evaluation modeled colonization as an event preceding CR-BSI (23), 4 considered these as mutually exclusive events (24,26,27,30), and 3 did not consider colonization (25,28,29). Two models included adverse events specific to the intervention (28,30), but this was not consistent across studies, with only 1 of the 3 evaluations of antiseptic-impregnated catheters including incidence of hypersensitivity reactions to the catheter (30). In 7 evaluations (23–26,28–30), only the outcomes that would arise during the period of hospitalization were included. In another evaluation (27), the time horizon described the patient’s lifetime.

Source and Incorporation of Data

Authors of all evaluations stated the baseline data used in the model along with its source; 5 had information in a table format (23,24,27,29,30). Most parameter estimates came from the published literature, although 5 evaluations performed their own cost calculations for the intervention (23–26,29) and 1 used original patient trial data for the estimates of daily incidence and relative risk for infectious events (27). Seven evaluations (23,25–30) discussed simplifying assumptions and issues of generalizability.

For 6 evaluations (23,26–30), the most important model parameters were identified (Table 5), with the following 3 parameters consistently important: reduction in risk for CR-BSI caused by the intervention, baseline incidence of CR-BSI, and cost of treating a CR-BSI. The ranks of evidence used for these and other model parameters are shown in Table 6. The level of evidence used for the effectiveness of the intervention was generally high, and authors of all evaluations provided information on how they selected the data used for this parameter. However, the level of evidence used for the cost and baseline incidence of CR-BSI was generally of lower quality; little detail was given in the reports of the evaluations as to why 1 particular estimate for a parameter was chosen over another. In particular, in all evaluations, reference was made in the introduction or discussion section to relevant information on the cost and deaths attributable to CR-BSI that was not used in the analysis. This explains the wide variation in the source and value of the estimates used for parameters between the evaluations (Table 5).

Model parameters were expressed as probability distributions for only 3 studies (23,27,30), even though this method provided an opportunity to appropriately describe parameter uncertainty. All 3 studies specified the choice of distribution for model parameters and the rationale for this choice. The remaining studies (24–26,28,29) used point estimates and a range for each parameter across which the estimate was varied in sensitivity analyses. Similar to the baseline estimates, no information was given on how ranges used for sensitivity analysis were decided upon, aside from a double-it and half-it approach.

Model Evaluation

All evaluations used deterministic sensitivity analyses by varying parameters across a range of point estimates either 1 at a time (1-way) or concurrently (multiway). Four studies (25,28–30) reported results of threshold analyses, i.e., the value of each parameter at which the conclusions from the analysis would change, and 6 studies (23,24,26,27,29,30) reported results of scenario analyses, i.e., results where all parameters are set to favor each specific intervention in turn (Table 2). The 3 evaluations that characterized parameters as distributions (23,27,30) also used probabilistic sensitivity analysis, which enabled calculation of confidence intervals around their point estimates of incremental costs and benefits.

In the 6 evaluations where the intervention was cost-saving (23,24,26,27,29,30), the conclusions were robust to the sensitivity analyses. In the 2 evaluations where an incremental cost-effectiveness ratio could be calculated (25,28), different conclusions were drawn in different scenarios (Table 3). Scenario analyses used in 6 evaluations (23,24,26,27,29,30) indicated internal consistency in the models, i.e., they behaved logically and as expected. However, only 1 evaluation (27) made an explicit statement on internal consistency about checks performed during the model construction and analysis. Authors of 7 evaluations discussed caveats to their work (23–27,29,30).

Usefulness of Evaluations

We have data on the cost-effectiveness of only 6 interventions. These interventions were evaluated separately and not compared with each other. Furthermore, many other interventions have been shown to be clinically effective but, there are no data on their cost-effectiveness. This finding is not consistent with current guidelines (2), which recommend that “it is logical to use multiple strategies concomitantly.” The 100,000 Lives Campaign is also formed on the basis of a group of interventions. The existing economic evidence is therefore incomplete and cannot be used to form a coherent policy for preventing CR-BSI. Infection control practitioners and other decision makers require information on the relative cost-effectiveness of all relevant groups of interventions rather than individual strategies (8). A good example of using cost-effectiveness to inform a complete policy is provided by Frazier et al (33). They evaluated 21 competing strategies for population-based colorectal cancer screening and included all relevant screening methods and frequencies. This study provides policymakers with complete information in as much as all available choices have been compared.

The failure to specify baseline values (i.e., the value authors believe is most likely) for model parameters is also problematic. Instead of estimating a baseline model and then testing whether the conclusions are robust to high and low values, some authors report all possible results on the basis of all possible values for some parameters. This shifts the responsibility of interpreting the results to the reader. The failure to describe how high and low values were chosen for key parameters (i.e., the double-it and half-it approach) compounds the problem.

Assessing Quality

There was a lack of transparency in the development of model structure. Model structure may have been driven by availability of data rather than careful review of the natural progression of the disease. This could undermine the external consistency of the evaluations as they appear to users. The choice of short-time horizons and narrow economic perspectives inhibits the usefulness of these evaluations by excluding relevant costs and health outcomes from the analysis. The current evidence may represent a blinkered view of the problem and how it should be managed. This situation in turn reduces the extent to which the value of infection control can be compared with other healthcare spending such as cardiac surgery and diabetes prevention.

The quality of data incorporated in the models is highly variable. The authors of 7 studies (23,24,26–30) suggest that their results are compromised by an absence of high-quality or precise information, often for key parameters in the model. This finding leads to some skepticism about the results (31). Researchers are attempting to provide better estimates of the health and economic outcomes attributable to CR-BSI (34). However, a model should not be criticized on the basis of the quality of data used per se. Rather, it should be judged on the techniques used to identify and incorporate the highest quality appropriate and relevant data possible (35) for all parameters, not just those relating to effectiveness. Given the lack of information provided by the authors about this process, a more systematic approach to selecting evidence needs to be introduced. Generic tools such as the hierarchy used here (19) are useful to judge evidence quality, but this may need to be supplemented with tools such as the hierarchy of quasi-experimental study designs, given the prevalence of the use of these designs in the infection control literature (36). Where multiple pieces of relevant information are available, techniques exist for the synthesis of diverse evidence (37).

Given the variations in data quality, selecting the best evidence and then propagating the effect of uncertainty in this evidence to the conclusions drawn are important. A good method is probabilistic sensitivity analysis (38). This method was used in 3 evaluations (23,27,30). This technique characterizes parameter estimates as distributions rather than discrete values and conducts multiple simulations of the model that draw different parameter values each time from the distributions. This enables the uncertainty around the costs and benefits of a given intervention to be described and the relative contribution to all uncertainty arising from each parameter to be estimated. The next step, which was not conducted for any evaluation, is to estimate the value of collecting more data to inform these parameters (39). This step would be particularly relevant to the key parameters identified in this review. The current methods used to derive estimates of costs and deaths attributable to CR-BSI are subject to some bias and may not make intuitive sense to clinicians (31). This issue is problematic because these methods are important components in the model, often driving the changes in costs and benefits, and it is likely this finding partly explains why so many interventions appear cost-saving.

This review has some limitations. Despite use of a broad search strategy, we may not have identified all model-based economic evaluations in this area; some evaluations may not have been published or are available only as abstracts. Also, our assessment of the quality of evaluations using the good practice criteria may reflect the way evaluations are reported rather than conducted. In fact, word limits often prevent authors from providing a full description of methods. However, any indication that a criterion was addressed was taken as an evaluation that met that attribute.