Hospital Compare Web Site May Mislead Consumers

PHILADELPHIA, Dec. 13 -- Performance measures for hospitals adopted by Medicare and the Joint Commission on Accreditation of Healthcare Organizations may only be loosely related to patient outcomes.

Hospitals performing in the 75th percentile on 10 performance measures had only slightly better mortality rates compared with institutions performing in the 25th percentile, reported Rachel M. Werner, M.D., Ph.D., and Eric T. Bradlow, Ph.D., both of the University of Pennsylvania here, in the Dec. 13 issue of the Journal of the American Medical Association.

The researchers compared performance ratings for 3,657 acute care hospitals on the Centers for Medicare & Medicaid Services' Web site Hospital Compare against hospital risk-adjusted mortality rates measured from Medicare Part A claims data.

"These findings should not undermine current efforts to improve health care quality through performance measurement and reporting," Drs. Werner and Bradlow wrote. "However, attention should be focused on finding measures of health care quality that are more tightly linked to patient outcomes. Only then will performance measurement live up to expectations for improving health care quality."

In a related commentary, Susan D. Horn, Ph.D., of the Institute for Clinical Outcomes Research in Salt Lake City, said the results of the analysis cast doubt on the measures' appropriateness as a basis for pay-for-performance systems or for consumers to select a high-quality hospital.

"If performance measures are not strongly associated with better outcomes," she wrote, "why should clinicians and health care centers be required to collect and submit the data, and why would payers and consumers want to act on them?"

For all acute myocardial infarction performance measures, the absolute reduction in these mortality rates between hospitals performing in the 25th and 75th percentiles was 0.005 for inpatient mortality, 0.006 for 30-day mortality, and 0.012 for one-year mortality (P<0.001 for each comparison).

For all heart failure performance measures, the absolute mortality reduction ranged from 0.001 for inpatient mortality (P=0.03) to 0.002 for one-year mortality (P=0.08). For the pneumonia performance measures, the absolute reduction ranged from 0.01 for 30-day mortality (P=0.05) to 0.005 for inpatient mortality (P<0.001).

Since the measures were only "modestly correlated" to mortality, they may not be clinical important, Drs. Werner and Bradlow said.

"Although hospital performance predicted differences in risk-adjusted mortality rates that were statistically significant in some cases, these differences were small," they wrote. "Based on these results, the ability of performance measures to detect clinically meaningful differences in quality across hospitals is questionable."

Since the proportion of hospitals reporting the 10 measures to the CMS reached 98% in 2004 with financial incentives provided by the Medicare Modernization Act, the researchers used data from that year for the study.

The CMS performance measures used included five measures of care for acute MI (aspirin use and beta-blocker use within 24 hours of arrival, angiotensin-converting enzyme inhibitor use for left ventricular dysfunction, aspirin prescribed at discharge, and beta-blocker prescribed at discharge), two measures for heart failure (left ventricular function assessment and ACE inhibitor use for left ventricular dysfunction), and three for pneumonia (timing of initial antibiotics, pneumococcal vaccination, and oxygenation assessment within 24 hours of admission).

Performance was calculated for each measure as the proportion of patients who received indicated care out of all the patients who were eligible for it. Hospitals with fewer than 25 patients eligible for a measure were excluded.

Then, the investigators used Medicare Part A claims data from Medicare Provider Analysis and Review (MEDPAR) database for 2004 to calculate hospital risk-adjusted mortality rates (the summed ratio of expected to observed mortality rates divided by total number of patients).

They found that the average risk-adjusted mortality rate ranged from 0.27 to 0.40. High-mortality hospitals were less likely to be large, for-profit, or teaching hospitals or to have open heart surgery capabilities (P?0.001 for all comparisons).

For hospitals performing in the 75th versus 25th percentiles on all measures within a condition, the effect size for mortality rates ranged from:

• 0.008 (P=0.06) to 0.018 (P=0.008) for acute MI,
• 0.003 (P=0.09) to 0.014 (P<0.001) for pneumonia, and
• -0.013 (P=0.06) to -0.038 (P=0.45) for heart failure.

For this "all-or-none" measure, 8% to 14% of hospitals qualified as high performing.

Based on an estimate of 750,000 patients hospitalized each year for acute MI, 3,000 lives could have been saved if a third of the patients treated at the lowest-performing hospitals had been treated at the highest-performing hospitals where mortality rates were 1.2% lower. However, number of lives at risk becomes smaller when access to high-performing hospitals is figured in, the researchers said. For example, only a third of Medicare beneficiaries live within 30 miles of a high-performing hospital in the southern states.

Reasons for the modest effect may include that the performance measures do not assess global quality of care at the hospitals and that mortality rates are likely influenced by many independent factors, such as electronic health record use, staffing levels, and activities of quality oversight committees. Also, there were small absolute differences between high- and low-performing hospitals. For example, hospitals performing in the 25th percentile give aspirin at admission for acute MI only 7% less often than those in the 75th percentile.

While the study included only Medicare beneficiaries, this population makes up more than half of the admissions for the conditions considered in the study and the findings are likely generalizable to the broader U.S. population, Drs. Werner and Bradlow said.

Rather than abandon quality measures based on the findings, the authors suggested further research to find clinically relevant measures.

In her commentary, Dr. Horn concluded that "in the real world where multiple clinical variables and patient factors affect outcomes," performance measures should be informed by comprehensive observational studies as well as randomized controlled trials.