An Approach to Measuring and Comparing Performance & Improvement in Hospitals by John S. Coldiron, M.D., M.P.H.

Monitoring, measuring and assessing performance as well as the success of planned performance improvements is vital to the decision-making and resource-allocation processes of any organization. Hospitals are given specific guidance regarding these activities through the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) standards established for leadership and for improving organizational performance. These standards obligate leadership to identify and select priorities and to improve organizational performance (1) and they confer the responsibility for collecting and analyzing data and for demonstrating performance improvement when necessary (2). To accomplish the mandated monitoring and improve performance, a hospital must have an infrastructure which provides the appropriate organizational framework, supporting policies and procedures, and a data collection and reporting system. This article will focus on data collection and reporting.

To accomplish the collection and reporting functions in the most efficient manner with the addition of a minimum of resources requires steps that lead to the following:

• Consolidation of all organizational indicators into a single database.
• Simplification of reporting of indicators by providing for direct, one-time, on-line data entry by the responsible person in a standardized format.
• Production of user-friendly reports that are specific to the various functional entities, e.g. departments, committees, teams, governing bodies, professional groupings, managers and external agencies, in a summary format that highlights the most useful and relevant data for that specific group. These reports should illustrate trends over time and, as appropriate, allow production of control charts so that performance variations are identified.

A schema for the overall design of the data collection and reporting system required to do these critical functions is illustrated in Diagram 1.

Diagram 1 Performance Evaluation and Improvement Database Relationships

Central to the data support system are the performance evaluation and improvement (PE&I) databases. As illustrated in Diagram 1, the sources of data are numerous, varying from organization to organization in both content and data-collection software used to collect the data. This PE&I reporting system summarizes key data elements from the selected sources. Also, this can data produce a variety of reports uniquely targeted for various internal and external audiences. An indicator module can be used to produce a dashboard of indicators specific to the areas of interest of any given sub-unit within the organization. Table 1 and Table 2 are examples of administrative and clinical "report cards" using the monitoring indicators selected by the organizational unit tracking the processes or outcomes.

Likewise a report can be developed that displays the progress for those indicators tracking improvement (Table 3).

The same data sources can be used to provide case-specific data in provider profile formats. For example, a physician profile that would be useful in the medical staff reappointment process(3) can be displayed.

Finally, by measuring the variation from the mean for monitoring indicators and calculating the mean variation of all indicators from their individual targets, an overall institutional measure of performance is developed. As well, the percent progress of the improvement projects toward the target can be calculated. This is illustrated by the Index Trend Chart:

These indices give a "one number" measurement of quality for the stable processes of the institution (monitoring indicators) as well as the relative success of the selected improvement efforts (improvement indicators). If truly representative of the scope of the institution's services and if validated properly, infra vida, these indices not only provide the measure of quality in the subject organization but also can be used for comparison to others.

The starting point for implementation of a system to measure performance through indicator profiles is insistence on institution-wide standardized input of data elements into each indicator. An example of this is shown in the illustration below:

In developing the indicator description, one must specify why the target is appropriate, and also, clearly describe the methodology for collection of data. The rationale for any adjustment for case mix needs to be described and applied to both the indicator and data used to establish the performance target. Finally, the indicators selected must truly reflect important aspects of the scope of services provided by the departments within the institution.

If steps are taken to assure that the indicators in the PE& I database(s) truly reflect the characteristics of key process variables and outcomes of the components of the scope of services of the institution, if it is validated that the targets established are suitable, if the methodology used to collect the data is verified as correct and if the reporting of the results is validated, then comparisons can be made from institution to institution. The issue of validation of the accuracy of the data collected is of great importance. Within the organization the quality improvement staff can serve as internal auditors; and, for external comparisons, validators sponsored by trade organizations, customer associations or government agencies, can perform the role.

Most institutions have a large number of indicators they generate for their own internal use or to meet requirements of an external licensing or certifying agency. Consolidating the indicator database does not mean these indicators should be abandoned; but rather just standardized in terms of their format and aggregated into one database that can be summarized in flexible, user-defined formats. The use of control charts to track trends and the statistical process control approach to the analysis of variation of performance over time provides the basis for focused review and possible action(4). This process can be facilitated by software such as QualaCare(5), which allows desktop entry of values by designated individuals throughout the organization. This approach to monitoring performance has proven successful in over three years of institution-wide use in an acute general community hospital, which includes psychiatric, rehabilitation and subacute services (Newport Hospital, Newport, RI).

In summary, it is possible, practical and desirable to evaluate and measure improvement of the functions and dimensions of performance within an organization through the use of an indicator reporting tool. The ability to aggregate, summarize and report data in user-defined formats allows identification of areas for focus and further investigation. It also produces an overall index of the stability of the organization's performance and the progress being made on improvement projects that can be compared to other organizations using the same methodology.

References

1. Comprehensive Accreditation Manual for Hospitals. Standard LD.4.3, Joint Commission on Accreditation of Healthcare Organization., One Renaissance Boulevard, Oakbrook Terrace, IL.

2. Comprehensive Accreditation Manual for Hospitals. Standard LD 4.5, Joint Commission on Accreditation of Healthcare Organization., One Renaissance Boulevard, Oakbrook Terrace, IL.

3. Coldiron, J. S., Using Physician Profiles in the Reappointment Process pp. 107-113 (First Edition), pp. 255-264 (Second Edition). The Physician Leader's Guide, Rockville, MD: Bader and Associates, Inc., 1992 and 1998.

4. Maleyeff, J. et. al, Analysis of Hospital Mortality for Continuous Improvement pp. 23-31. Journal of Healthcare Risk Management, Spring 2001.

5. Insight Health Solutions. 225 Chapman Street, Providence, RI 02905-4507

John S. Coldiron, MD, MPH, is Vice President for Medical Management at Insight Health Solutions, Inc.

e-mail: jcoldiron@insighthealthsolutions.com

This article was also published in Medicine & Health Rhode Island

Vol. 84 | No. 11 | November 2001 | pp. 356-360

Medicine & Health Rhode Island is one of the oldest established state Medical Journals. It has appeared on a monthly basis since 1904. This publication is under the joint editorial sponsorship of the Brown University School of Medicine, Rhode Island Department of Health, and the Rhode Island Medical Society.