By Tuyen Tran, M.D.
Since the Institute of Medicine’s (IOM) report in 1999, To Err is Human, multiple stakeholders (government agencies, accreditation bodies, payers, hospitals, providers, and the public) have responded with various metrics to monitor and interventions to implement. They wanted to improve the healthcare and positively effect changes to prevent the annual avoidable 98,000 deaths and 1 million injuries related to medical errors1. Stakeholders demanded positive outcomes and they wanted providers accountable. As the chanting of “First, do no harm!” continued, hospitals and providers became mired in performance measures and report cards. But, are these metrics valid?
Mortality is comprised of three variables: 1) patient risk-factors (case-mix), 2) random chance, and 3) quality of care. Despite the best attempt at risk-adjustment, the calculations cannot account for unmeasured and/or immeasurable factors. That is, the outputs of risk adjustment regression models depend upon the input variables, which can vary widely. For example, comorbidities (morbid obesity, dementia, and heart failure, level of frailty and disability) which impact mortality estimates are inconsistently documented.
As early as 1847, Ignaz Semmelweis observed that women delivered by physicians and medical students had a much higher post-delivery mortality than midwives (13-18% vs 2% respectively)2. (The differences were related to lack of hand washing.) Interest in comparing outcomes among healthcare providers certainly predates the IOM report. Of particular interest is the adjusted standardized mortality ratio (SMR, also known by various other names such as Risk-Adjusted Hospital Mortality Ratio). It is a comparison of the observed number of in-hospital deaths to the number expected based upon the hospital’s case mix. Ratios greater than 1 suggest unsafe healthcare and ratios less than 1 suggest safe practices. The thought process for “risk-adjustment” is that if the contributions from patient case-mix factors are removed, the residual unexplained variation is related to quality of care. This is perfect! First, the end point is concrete and everyone would consider the outcome of death important. Second, the information is readily available from most administrative databases. Third, the ratio allows for inter-hospital comparisons. Finally, the data are amenable to statistical analyses and graphical displays to facilitate interpretation.
There is the inevitable desire to conclude that a favorable SMR (ratio < 1) indicates a safe hospital, or vice versa. Death in modern hospitals is relatively rare (5-10%), and forensic clinical analyses of these deaths show that only 5% are attributable to unsafe care3. Thus, mathematically, due to the low rates of occurrence, only 8% of hospitals with unfavorable risk adjusted SMR (ratio >1) will truly be more “unsafe” than the average hospital. On the other hand, of the hospitals with favorable SMR, 10 out of 11 of these hospitals may actually be more “unsafe” than the average hospital. The reason is that most quality issues may result in injury or prolonged hospital stays; but, they do not cause death4. Most unsafe practices do not cause death and most deaths are not the result of unsafe care.
Of course, one preventable injury is one too many! Despite the tremendous allocation of resources into the improvement of quality and safety, the fact is that patients are needlessly harmed as a complication of receiving healthcare5. Unfortunately, there are no reliable tools to accurately measure the quality of care for physicians, hospitals, or populations. And until these valid metrics are found, tension will continue to mount among those who seek safe quality care (public), those who have the obligation to protect the public (policy makers), and the providers who do wish to deliver safe and quality healthcare.
Kohn LT, Corrigan JM, Donaldson MS, eds. To err is human: building a safer health system. Washington, DC: National Academies Press, 1999.
Best M, Neuhauser D. Ignaz Semmelweis and the birth of infection control. Qual Saf Health Care 2004; 13:233-4.
Scott IA, Brand CA, Phelps GE, Barker AL, Cameron PA. Using hospital standardized mortality ratios to assess quality of care – proceed with extreme caution. Med J Aust. 2011;194:645-648
Girling AJ, Hofer TP, Wu J, et al. Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modeling study. BMJ Qual Saf. 2012;21:1052-1056.
Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med. 2010;363(22):2124-34.