September 29, 2022

Patients admitted to inpatient facilities are at risk for acute physiologic deterioration. This can lead to prolonged hospitalization, admission to the ICU, and even cardiorespiratory arrest (1,2). Worsening in a patient’s clinical condition often remains undetected for hours prior to escalation of care (1). Attempts at recognizing deterioration early have been developed and range from simple alerts based on vital sign alterations to trend analysis and complex early warning scores (EWS) (3). These efforts have been combined with multidisciplinary rapid response teams (RRTs) aimed at timely intervention and prevention of cardiorespiratory arrest (2). Owing partly to limitations of the aggregate risk scores and partly to variable RRT availability and composition, these efforts have not led to consistent improvements in outcomes (4).

Health information technology (HIT) is broadly defined as the incorporation of various information sources, data, and technology to facilitate improved communication and decision-making (5). The widespread implementation of electronic medical records (EMRs) has allowed access to larger quantities of clinical data and utilization of prediction analytics (6). EMR-based alarms have emerged to support timely detection of acute conditions such as sepsis, acute kidney injury (AKI), and respiratory failure (7–9). Digital clinical decision support has also been created to help standardize the approach and management of deteriorating patients (10).

Although a recent meta-analysis reported EMR improved patient safety by reducing medication errors and adverse drug reactions, that study did not reveal any improvement in mortality (11). Another meta-analysis focused on a broad range of HIT in the inpatient setting and did not demonstrate reduction in hospital mortality or length of stay (LOS) either (12).

Much work has been done to develop systems identifying actionable deterioration when a patient may benefit from early attention and action from clinicians. However, patient outcomes from HIT supporting early detection of patients with actionable worsening conditions remain unknown. The objective of this systematic review (SR) and meta-analysis was to evaluate the impact of HIT for early detection of patient deterioration on patient mortality and LOS in the acute care hospital setting. This systematic evaluation may help clinicians and institutions to make informed decisions about the utilization and implementation of HIT within process and workflow systems across acute care clinical settings.

MATERIALS AND METHODS

The results of the study were reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statements (13) (Supplemental Table 1, https://links.lww.com/CCM/H90). The Covidence software (Veritas Health Innovation, Melbourne, Australia) was used for data collection (14).

Data Sources and Search Strategy

A comprehensive search of several databases from 1990, when robust information technology infrastructure in hospitals became more widespread, to January 19, 2021, was conducted. The databases included MEDLINE and Epub Ahead of Print, In-Process & Other Non-Indexed Citations and Daily, Embase, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, and Scopus. The search strategy was designed and conducted by an experienced librarian with input from study investigators. Controlled vocabulary supplemented with keywords was used to search for studies of interest. The actual strategy listing all search terms used and how they were combined is available in Supplemental Table 2 (https://links.lww.com/CCM/H91). The additional sources included gray literature search and reference mining.

Study Selection

We included studies that enrolled patients hospitalized on inpatient floors, in ICU, or evaluated in the emergency department (ED). Eligible studies assessed HIT for early detection of and notification about patients experiencing deterioration or at high risk of deterioration, as an intervention. Comparison groups received usual care in the same study settings. Eligible studies reported at least one end point of interest: hospital LOS, ICU LOS, or mortality at any time point.

We excluded studies that used an HIT intervention not detecting deterioration, and developed or validated an HIT intervention only without implementation into practice.

Titles, abstracts, and full texts of identified studies were independently reviewed by pairs of reviewers (S.H., K.L., Y.P., H.L., A.T., A.K.B.) using prespecified eligibility criteria. Disagreements were resolved by a third reviewer (S.H., V.H.) or through group discussion to reach consensus.

Data Extraction

Study details of included articles were abstracted by two independent reviewers (K.L., Y.P., H.L., A.T.) using a standardized data extraction form. Additional reviewers (S.H., A.K.B.) resolved disagreements. Data abstracted included study timeline, setting, population and size, intervention description, and outcomes (Supplemental Appendix 1, https://links.lww.com/CCM/H92).

Outcome Measures

The primary outcome was difference in hospital mortality between the intervention and comparison groups. The secondary outcomes were hospital LOS, ICU LOS, ICU mortality, and mortality at other commonly reported time points. All outcomes were prespecified.

Data Synthesis and Analysis

When possible, we extracted or calculated the odds ratios (OR) and corresponding 95% CIs for binary outcomes (mortality). We used adjusted OR when available. For continuous outcomes (LOS), we calculated mean differences (MDs) using 95% CIs.

The DerSimonian and Laird random effect method was used for quantitative synthesis of data when at least three eligible articles included the desired outcome. Meta-analyses were performed separately for randomized controlled trials (RCTs) and for pre-post studies.

We evaluated heterogeneity between studies using the I2 statistics. I2 0–30% was categorized as low heterogeneity, 31–60% as moderate, and greater than 60% as substantial heterogeneity (15).

To explore potential sources of heterogeneity, we conducted predetermined subgroup analyses based on the study setting (ED, hospital floor, or ICU), type of patient deterioration identified by HIT (sepsis, AKI, and others), and risk of bias (ROB).

We also conducted post hoc analysis of RCTs to assess possible changes in the cumulative evidence about the effect of HIT on hospital mortality over time. Sensitivity analyses were conducted to assess robustness of the synthesized results. Analyses were performed using OpenMeta Analyst—an open-source, cross-platform software for advanced meta-analysis (16). Two-tailed p value of less than 0.05 was considered statistically significant.

Risk of Bias/Quality Assessment

The ROB was assessed by Drs. Herasevich and Herasevich using the Revised Cochrane ROB tool for randomized trials (17) and The Risk Of Bias In Nonrandomized Studies—of Interventions assessment tool (18).

We evaluated the strength of evidence using Grading of Recommendations Assessment, Development, and Evaluation approach (19). Per standard grading, evaluation of RCTs was initially considered as high quality of evidence and observational studies as low quality of evidence. To assess potential modifying factors affecting the strength of evidence, we evaluated methodological limitations of included studies, precision, directness, consistency, and publication bias (19).

RESULTS

Study Selection

The search strategy identified 2,767 studies with 44 additional studies identified through additional searches (20). After removing duplicates, 2,810 papers were screened using titles/abstracts. Following screening, 2,552 abstracts were removed, and 258 papers remained for full-text review. Among the final set of 30 studies, 21 contained quantitative data and were included in the meta-analyses for one or more outcomes. See PRISMA diagram (Fig. 1) for study selection, phases, and reasons for exclusion.

F1
Figure 1.:

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram.

Eligible Studies and Participant Characteristics

Supplemental Table 3 (https://links.lww.com/CCM/H93) summarizes the characteristics of the 30 eligible studies. Most of the studies were conducted in the United States, six were conducted in Europe, and two in Asia. Twenty-three studies were single center, and seven were multicenter.

Eighteen studies were based on hospital floors (6,10,21–36) and five in ICU (37–41). Two studies examined HIT implementation in the ED (42,43) and then analyzed outcomes among those hospitalized following ED presentation. Five studies were based on both ICU and the hospital floor (44–48).

Seven studies were RCTs, including two cluster-randomized trials (22,41) and five individually randomized trials (29,39,46–48). Twenty-three studies used a pre- and a postimplementation design (6,10,21,23–28,30–38,40,42–45).

Seven studies evaluated HIT for detection of AKI (21,32,37,38,46–48), 10 were designed for early detection of sepsis or systemic inflammatory response syndrome (10,24,25,27,33,35,39,42–44), and the remaining 13 studies for other types of deterioration (6,22,23,26,28–31,34,36,40,41,45) such as respiratory or other physiologic deterioration. There was a lack of uniformity in how deterioration was quantified with some investigators using scores or criteria for clinical syndromes and some using changes in vital signs, but all used robust approaches to define deterioration (Supplemental Table 4, https://links.lww.com/CCM/H94).

Baseline characteristics in intervention and comparison groups in included studies were similar. The median study duration was 1.5 years with wide variation from 2.5 months to 12 years.

Outcome Measures

Some studies assessed outcomes of interest among the entire study cohort, whereas other studies only assessed the outcomes among those patients meeting the criteria for deterioration both in intervention and comparison groups. Thus, we conducted two types of meta-analyses: one evaluating the mortality and hospital LOS for all included study patients (entire study cohort) and one evaluating only those patients who reached the alert threshold defined for each study and, therefore, detectable by the HIT.

All outcomes for eligible studies are summarized in Supplemental Table 4 (https://links.lww.com/CCM/H94). However, we limited our analysis to the primary and secondary outcomes described above. We conducted separate meta-analyses for RCTs and pre-post studies for each outcome.

Risk of Bias/Quality Appraisal

Among the RCTs, the overall ROB was low or moderate for most studies due to lack of blinding among clinicians and outcome assessors (Supplemental Table 5, https://links.lww.com/CCM/H95). In the pre-post studies, ROB was moderate or high for most studies due to potential confounding and incomplete reporting of study results (Supplemental Table 6, https://links.lww.com/CCM/H96).

Pooled effect size and quality of evidence for hospital mortality and LOS are reported in Supplemental Table 7 (https://links.lww.com/CCM/H97). The quality of evidence of included studies was low due to methodological limitations, inconsistency, and imprecision.

Mortality

All included studies assessed mortality as an outcome, although at different time points.

Hospital Mortality.

Twenty-eight of the 30 studies (6,10,21–27,29–45,47,48) reported hospital mortality. Sixteen studies assessing hospital mortality were evaluated in the meta-analyses. Of these, 11 (6,30,31,34,37,39–41,43,45,47) reported hospital mortality for the entire study cohort, two studies (33,42) reported the outcome only for those patients meeting deterioration criteria, and three (21,22,35) reported both.

Entire Cohort.

In the meta-analysis of four RCTs, the implementation of HIT for early detection of patient deterioration was not associated with a significant decrease in hospital mortality (OR, 0.99 [95% CI, 0.80–1.21]) (Fig. 2).

F2
Figure 2.:

Meta-analyses on hospital mortality in patients who received the intervention (Health Information Technology for early detection of deterioration) compared with usual care. Entire study cohort. A, Randomized controlled trials. B, Nonrandomized (pre-post) studies. C, Sensitivity analysis of the pre-post studies. The size of the data markers represents the weight each study has in the pooled result.

Heterogeneity within this subset of studies was moderate and can be partially explained by the difference in types of deterioration detected by HIT.

The meta-analysis of 10 pre-post studies demonstrated a significant association between the use of HIT and improved mortality (OR, 0.78 [95% CI, 0.70–0.87]) (Fig. 2). The heterogeneity was moderate in this group and may be attributed to the difference in types of deterioration detected (Supplemental Fig. 1, https://links.lww.com/CCM/H98; legend, https://links.lww.com/CCM/H92). Sensitivity analysis demonstrated the stability of the pooled effect size and only a marginal improvement in heterogeneity (Fig. 2).

Study Participants Meeting Criteria for Deterioration.

Implementation of HIT was not associated with a statistically significant decrease in hospital mortality in three RCTs (22,29,48). Meta-analysis was not performed as one study did not include sufficient data.

Meta-analysis of five pre-post studies demonstrated a significant association between HIT and a decrease in hospital mortality (OR, 0.92 [95% CI, 0.87–0.97]) (Fig. 3). The heterogeneity within this subset was low.

F3
Figure 3.:

Meta-analysis on hospital mortality in patients who met the criteria for deterioration among those who received the intervention (Health Information Technology for early detection of deterioration) compared with usual care. Pre-post studies. The size of the data markers represents the weight each study has in the pooled result.

Additional mortality outcomes are reported in Supplemental Appendix 1 (https://links.lww.com/CCM/H92).

Hospital LOS

Twenty-three of the 30 included studies assessed hospital LOS as an outcome (6,10,21–23,25–31,33–37,39–41,44,46,47). Sixteen studies included quantitative data for evaluation in the meta-analysis. Of these, 11 (26,27,30,31,34,37,39–41,46,47) reported the hospital LOS for the entire study cohort, three studies (23,29,33) reported the hospital LOS only for those patients who met the criteria for deterioration, and four (21,22,25,35) reported both.

Entire Cohort

In the meta-analysis of five RCTs, no significant difference in hospital LOS was found (MD, 0.10 [95% CI, –0.07 to 0.27]) (Fig. 4). The heterogeneity was low in this group of studies.

F4
Figure 4.:

Meta-analyses on hospital length of stay in patients who received the intervention (Health Information Technology for early detection of deterioration) compared with usual care. Entire study cohort. A, Randomized controlled trials. B, Nonrandomized (pre-post) studies. C, Sensitivity analysis of the pre-post studies. The size of the data markers represents the weight each study has in the pooled result.

Meta-analysis of 10 pre-post studies demonstrated significant association of HIT with reduced LOS (MD, –0.29 [95% CI, –0.51 to –0.07]) (Fig. 4). However, the heterogeneity in this set of studies was substantial and could not be fully explained by difference in ROB, study settings, or types of detected deterioration (Supplemental Fig. 2, https://links.lww.com/CCM/H99; legend, https://links.lww.com/CCM/H92). One obvious outlier, Olchanski et al (40), compared two cohorts with time difference in 4 years, and its results were likely affected by the practice changes over time. Sensitivity analysis showed that following removal of this study, no significant association between HIT and improvement in hospital LOS was demonstrated (MD, –0.15 [95% CI, –0.33 to 0.03]) (Fig. 4).

Study Participants Meeting Criteria for Deterioration.

Two RCTs evaluating hospital LOS among patients meeting criteria for deterioration (22,29) did not demonstrate significant improvement in LOS.

However, in the meta-analysis of four pre-post studies, HIT implementation was associated with a significant reduction in hospital LOS (MD, –0.29 [95% CI, –0.48 to –0.11]) (Fig. 5).

F5
Figure 5.:

Meta-analysis on hospital length of stay in patients who met the criteria for deterioration among those who received the intervention (Health Information Technology for early detection of deterioration) compared with usual care. Pre-post studies. The size of the data markers represents the weight each study has in the pooled result.

Additional LOS outcomes are reported in Supplemental Appendix 1 (https://links.lww.com/CCM/H92), Supplemental Figure 3 (https://links.lww.com/CCM/H100; legend, https://links.lww.com/CCM/H92), and Supplemental Figure 4 (https://links.lww.com/CCM/H101; legend, https://links.lww.com/CCM/H92).

DISCUSSION

In this SR and meta-analyses, we evaluated the impact of HIT for early detection of patient physiologic deterioration on hospital mortality and LOS. We included 30 studies assessing patients in acute care hospital settings. There was variability in setting, interventions, type of deterioration detected, and outcome measurement approaches. We conducted multiple analyses to compare similar study designs and groups with similar outcome approaches (studies reporting outcomes for entire study cohorts and only for patients meeting deterioration criteria).

We found that HIT for early detection of patient deterioration was not associated with a reduction in hospital mortality or LOS in the RCTs and associated meta-analyses. In the meta-analyses of pre-post studies, HIT intervention was significantly associated with improved hospital mortality and hospital LOS. ICU LOS did not change significantly with HIT interventions. LOS can be a challenging outcome measure due to competing risk of mortality and the potential of including those with a short survival time who have died (49).

There have been several SRs and meta-analyses exploring the impacts of HIT on patient outcomes. However, those studies differed from our study in several ways. The study by Varghese et al (50) focused on computerized decision support system (DSS) implementations and found positive but not clinically important improvements in patient outcomes. That study noted a lack of rigorous RCTs to assess clinical decision support. The SR by Despins (51) focused on detection of sepsis only and noted that current performance variability affected the impact on patient outcomes. Two more SRs have focused on a broad range of HIT including EMR, DSS, computerized physician order entry, and surveillance systems (“sniffers”) and have not demonstrated improvements in hospital mortality or LOS (11,12). In contrast to other SRs, we focused on the subset of HIT specifically designed for early detection of deterioration that had been implemented in acute care settings.

A notable finding of our work overall is the difference between the conclusions of the RCTs and the pre-post studies. HIT implementation was not associated with improvements in hospital mortality in the RCTs that are considered the gold standard of research and a rigorous approach to avoid confounding (52). The studies supporting the use of HIT were generally pre-post studies, and the conclusions from these studies need to be considered carefully due to the high probability of confounding outlined below.

We identified several categories of potential cofounders that may have played an important role in the improved outcomes in the pre-post studies in our SR. These were: 1) training and education of staff (6,37), 2) broad quality improvement projects in which the HIT was just one component (10,23,43), 3) change management assessments and general improvements over time (30,35,40,45), 4) complex multicomponent or multifaceted interventions that also included DSS and dashboards (40,45), and 5) the Hawthorne effect (10,23,34,37,53).

Although studies often reported that there were no known significant changes in the clinical practice during the study period, they were likely still prone to bias and influenced by time and overall improvements in practice. For example, one of the two studies demonstrating the highest benefit of HIT on hospital mortality (45) evaluated COVID patients early in pandemic, and it is likely that improvement in mortality was due to advances in COVID patient management rather than to HIT implementation (54). Another study compared a postimplementation cohort with historical controls from four years prior to implementation (40).

Undoubtedly, the mechanism by which HIT was integrated to the clinicians’ workflow is important. However, similar approaches to HIT integration may yield different results. Six studies in this SR evaluated HIT implementation to supplement RRT activations in settings where RRT activations were the standard of care. Of those, three pre-post studies demonstrated a decrease in hospital mortality associated with HIT intervention (23,30,34). Possible factors associated with the positive effect on hospital mortality included off-site nurse review to filter alerts before contacting the RRT, alerting bedside staff as well as RRT members, and probable improvements in practice over a prolonged study period. The other three studies (two pre-post studies that evaluated sepsis-related outcomes, and one RCT) did not demonstrate any significant improvement in hospital mortality (24,29,35).

Our SR has several strengths. We performed meta-analyses of studies reporting meaningful patient outcomes: LOS and mortality, rather than more immediately and easily measurable surrogate markers such as time to RRT activation, ICU transfer, or specific interventions, which helped us form robust conclusions (55). Study settings included all relevant acute care hospital populations: floor, ICU, and ED, and most studies were large. We only included studies assessing HIT that had been implemented in practice as opposed to studies that described development or validation of an HIT to assess “real-world” use of HIT and its effects on patient outcomes (41). Although our SR includes a broad range of settings and populations, we hoped this work would provide relevant insights across the spectrum of acute care.

Important limitations are as follows. Heterogeneity was found to be moderate or substantial in the meta-analyses of the studies evaluating hospital mortality, hospital, and ICU LOS among the entire study cohorts (Figs. 2 and 4; Supplemental Fig. 1, https://links.lww.com/CCM/H98; Supplemental Fig. 2, https://links.lww.com/CCM/H99; Supp lemental Fig. 3, https://links.lww.com/CCM/H100; Supplemental Fig. 4, https://links.lww.com/CCM/H101 [legend, https://links.lww.com/CCM/H92]). This heterogeneity was mostly attributed to the difference in types of deterioration detected and study flaws related to temporal and practice changes. HIT for the detection of patient deterioration included distinct types of electronic systems using data from continuous bedside monitoring, EMR, and other electronic documentation. There was not a uniform definition of criteria for clinical deterioration across all studies. The most common conditions identified were AKI, early sepsis, or physiologic deterioration based on EWS or vital signs parameters. The difference in baseline states and standards of care across study settings may also affect the effect of HIT implementation.

However, although type of deterioration, modality of assessment, and disease states differed, all HIT implementation mechanisms required emergent responses by the clinical team as an integral part of the intervention and were designed to alert the teams to deterioration earlier than usual practice.

The certainty of evidence of the included studies was low, mostly due to methodological limitations and inconsistency. Some studies described unadjusted results, and some results were imprecise including wide CIs. Therefore, it is possible that other unmeasured factors influenced the effectiveness of the intervention, potentially under- or overestimating the true impact.

Improved results after HIT implementation in the pre-post studies may be attributed more to practice advances and quality improvement initiatives rather than to HIT implementation itself.

CONCLUSIONS

In this SR and meta-analysis, the implementation of HIT for early detection of deterioration in acute care settings was not significantly associated with improved mortality or LOS in the meta-analyses of RCTs. In the meta-analyses of pre-post studies, HIT was associated with improvement in hospital mortality and hospital LOS; however, these results should be interpreted with caution. We believe the differences in patient outcomes between the findings of the RCTs, and pre-post studies may be secondary to multiple potential confounding factors including practice advances and quality improvement initiatives rather than to HIT implementation itself.

REFERENCES

1. Hillman KM, Bristow PJ, Chey T, et al.: Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med. 2002; 28:1629–1634

2. Lyons PG, Edelson DP, Churpek MM, et al.: Rapid response systems. Resuscitation. 2018; 128:191–197

3. Downey CL, Tahir W, Randell R, et al.: Strengths and limitations of early warning scores: A systematic review and narrative synthesis. Int J Nurs Stud. 2017; 76:106–119

4. Hillman K, Chen J, Cretikos M, et al.; MERIT study investigators: Introduction of the medical emergency team (MET) system: A cluster-randomised controlled trial. Lancet. 2005; 365:2091–2097

5. Alotaibi YK, Federico F, et al.: The impact of health information technology on patient safety. Saudi Med J. 2017; 38:1173–1180

6. Evans RS, et al.: Electronic health records: Then, now, and in the future. Yearb Med Inform. 2016; 25 (Suppl 1):S48–S61

7. Fletcher GS, Aaronson BA, White AA, et al.: Effect of a real-time electronic dashboard on a rapid response system. J Med Syst. 2017; 42:5

8. Huh JW, Lim CM, Koh Y, et al.: Activation of a medical emergency team using an electronic medical recording-based screening system*. Crit Care Med. 2014; 42:801–808

9. Lee SH, Lim CM, Koh Y, et al.: Effect of an electronic medical record-based screening system on a rapid response system: 8-years’ experience of a single center cohort. J Clin Med. 2020; 9:383

10. Manaktala S, Claypool SR, et al.: Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inform Assoc. 2017; 24:88–95

11. Campanella P, Lovato E, Marone C, et al.: The impact of electronic health records on healthcare quality: A systematic review and meta-analysis. Eur J Public Health. 2016; 26:60–64

12. Thompson G, O’Horo JC, Pickering BW, et al.: Impact of the electronic medical record on mortality, length of stay, and cost in the hospital and ICU: A systematic review and metaanalysis. Crit Care Med. 2015; 43:1276–1282

13. Page MJ, McKenzie JE, Bossuyt PM, et al.: The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ. 2021; 372:n71

14. Covidence Database. Melbourne, Australia, Veritas Health Innovation. Available at: https://www.covidence.org/. Accessed March 1, 2022

15. Higgins JP, Thompson SG, Deeks JJ, et al.: Measuring inconsistency in meta-analyses. BMJ. 2003; 327:557–560

16. OpenMetaAnalyst: Wallace BC, Dahabreh IJ, Trikalinos TA, et al.: Closing the gap between methodologists and end-users: R as a computational back-end. J Stat Softw. 2012; 49: 5. Available at: https://www.cebm.brown.edu/openmeta/download.html. Accessed March 1, 2022

17. Sterne JAC, Savović J, Page MJ, et al.: RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ. 2019; 366:l4898

18. Sterne JA, Hernán MA, Reeves BC, et al.: ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016; 355:i4919

19. Murad MH, et al.: Clinical practice guidelines: A primer on development and dissemination. Mayo Clin Proc. 2017; 92:423–433

20. Horsley T, Dingwall O, Sampson M, et al.: Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011; 2011:Mr000026

21. Al-Jaghbeer M, Dealmeida D, Bilderback A, et al.: Clinical decision support for in-hospital AKI. J Am Soc Nephrol. 2018; 29:654–660

22. Bailey TC, Chen Y, Mao Y, et al.: A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med. 2013; 8:236–242

23. Escobar GJ, Liu VX, Schuler A, et al.: Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med. 2020; 383:1951–1960

24. Fogerty RL, Sussman LS, Kenyon K, et al.: Using system inflammatory response syndrome as an easy-to-implement, sustainable, and automated tool for all-cause deterioration among medical inpatients. J Patient Saf. 2019; 15:e74–e77

25. Giannini HM, Ginestra JC, Chivers C, et al.: A machine learning algorithm to predict severe sepsis and septic shock: Development, implementation, and impact on clinical practice. Crit Care Med. 2019; 47:1485–1492

26. Heller AR, Mees ST, Lauterwald B, et al.: Detection of deteriorating patients on surgical wards outside the ICU by an automated MEWS-based early warning system with paging functionality. Ann Surg. 2020; 271:100–105

27. Horton DJ, Graves KK, Kukhareva PV, et al.: Modified early warning score-based clinical decision support: Cost impact and clinical outcomes in sepsis. JAMIA Open. 2020; 3:261–268

28. Huff S, Stephens K, Whiteman K, et al.: Implementation of a vital sign alert system to improve outcomes. J Nurs Care Qual. 2019; 34:346–351

29. Kollef MH, Chen Y, Heard K, et al.: A randomized trial of real-time automated clinical deterioration alerts sent to a rapid response team. J Hosp Med. 2014; 9:424–429

30. Kollef MH, Heard K, Chen Y, et al.: Mortality and length of stay trends following implementation of a rapid response system and real-time automated clinical deterioration alerts. Am J Med Qual. 2017; 32:12–18

31. Mestrom E, De Bie A, Steeg MV, et al.: Implementation of an automated early warning scoring system in a surgical ward: Practical use and effects on patient outcomes. PLoS One. 2019; 14:e0213402

32. Park S, Baek SH, Ahn S, et al.: Impact of electronic acute kidney injury (AKI) alerts with automated nephrologist consultation on detection and severity of AKI: A quality improvement study. Am J Kidney Dis. 2018; 71:9–19

33. Sawyer AM, Deal EN, Labelle AJ, et al.: Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011; 39:469–473

34. Subbe CP, Duller B, Bellomo R, et al.: Effect of an automated notification system for deteriorating ward patients on clinical outcomes. Crit Care. 2017; 21:52

35. Umscheid CA, Betesh J, VanZandbergen C, et al.: Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med. 2015; 10:26–31

36. Weller RS, Foard KL, Harwood TN, et al.: Evaluation of a wireless, portable, wearable multi-parameter vital signs monitor in hospitalized neurological and neurosurgical patients. J Clin Monit Comput. 2018; 32:945–951

37. Bourdeaux C, Ghosh E, Atallah L, et al.: Impact of a computerized decision support tool deployed in two intensive care units on acute kidney injury progression and guideline compliance: A prospective observational study. Crit Care. 2020; 24:656

38. Colpaert K, Hoste EA, Steurbaut K, et al.: Impact of real-time electronic alerting of acute kidney injury on therapeutic intervention and progression of RIFLE class. Crit Care Med. 2012; 40:1164–1170

39. Hooper MH, Weavind L, Wheeler AP, et al.: Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012; 40:2096–2101

40. Olchanski N, Dziadzko MA, Tiong IC, et al.: Can a novel ICU data display positively affect patient outcomes and save lives? J Med Syst. 2017; 41:171

41. Pickering BW, Dong Y, Ahmed A, et al.: The implementation of clinician designed, human-centered electronic medical record viewer in the intensive care unit: A pilot step-wedge cluster randomized trial. Int J Med Inform. 2015; 84:299–307

42. Berger T, Birnbaum A, Bijur P, et al.: A computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010; 1:394–407

43. Gatewood MO, Wemple M, Greco S, et al.: A quality improvement project to improve early sepsis care in the emergency department. BMJ Qual Saf. 2015; 24:787–795

44. McCoy A, Das R, et al.: Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual. 2017; 6:e000158

45. Vizcaychipi MP, Shovlin CL, McCarthy A, et al.; Gary Davies on behalf of the ChelWest COVID19 Consortium: Increase in COVID-19 inpatient survival following detection of Thromboembolic and Cytokine storm risk from the point of admission to hospital by a near real time Traffic-light System (TraCe-Tic). Braz J Infect Dis. 2020; 24:412–421

46. Wilson FP, Martin M, Yamamoto Y, et al.: Electronic health record alerts for acute kidney injury: Multicenter, randomized clinical trial. BMJ. 2021; 372:m4786

47. Wilson FP, Shashaty M, Testani J, et al.: Automated, electronic alerts for acute kidney injury: A single-blind, parallel-group, randomised controlled trial. Lancet. 2015; 385:1966–1974

48. Wu Y, Chen Y, Li S, et al.: Value of electronic alerts for acute kidney injury in high-risk wards: A pilot randomized controlled trial. Int Urol Nephrol. 2018; 50:1483–1488

49. Brock GN, Barnes C, Ramirez JA, et al.: How to handle mortality when investigating length of hospital stay and time to clinical stability. BMC Med Res Methodol. 2011; 11:144

50. Varghese J, Kleine M, Gessner SI, et al.: Effects of computerized decision support system implementations on patient outcomes in inpatient care: A systematic review. J Am Med Inform Assoc. 2018; 25:593–602

51. Despins LA, et al.: Automated detection of sepsis using electronic medical record data: A systematic review. J Healthc Qual. 2017; 39:322–333

52. Hariton E, Locascio JJ, et al.: Randomised controlled trials – the gold standard for effectiveness research: Study design: Randomised controlled trials. BJOG. 2018; 125:1716–1716

53. Sedgwick P, Greenwood N, et al.: Understanding the Hawthorne effect. BMJ. 2015; 351:h4672

54. Huang C, Soleimani J, Herasevich S, et al.: Clinical characteristics, treatment, and outcomes of critically ill patients with COVID-19: A scoping review. Mayo Clin Proc. 2021; 96:183–202

55. Ioannidis JP, Lau J, et al.: Pooling research results: Benefits and limitations of meta-analysis. Jt Comm J Qual Improv. 1999; 25:462–469