Data Quality Audit: Understanding Coding Variations

Helen Whittome and Gail Crook

Introduction

The Canadian Health Information Management Association (CHIMA) and the Ontario Health Information Management Association (OHIMA), with technical support from the Canadian Institute for Health Information (CIHI), conducted a pilot data quality audit (the Audit) of Ontario acute inpatient clinical data. This study was commissioned by the Ontario Ministry of Health and Long Term Care (MoHLTC) to help increase the MoHLTC's understanding of observed variations in hospital case mix and to review coding practices in a sample of Ontario hospitals.

The Audit was conducted in four phases by a team of trained re-abstractors. The professional abstractors were assigned to 10 Ontario hospitals to conduct on-site patient record re-abstraction. The re-abstracted clinical data were compared to the originally abstracted data. The study design included clinical data from the Discharge Abstract Database (DAD) from the 2001/2002 fiscal years. In total, 1,398 records from 5 Case Mix Groups (CMG(TM)) were re-abstracted. Each hospital had between 124 and 151 charts re-abstracted.

Where discrepancies between the original and re-abstracted data were observed, the re-abstractors assigned to each a discrepancy type and a reason code. The impact of the discrepancies on CMG assignment and Major Clinical Category (MCC) assignment, weighted cases (RIW(TM)), and expected length of stay (ELOS) were also measured.

Overall, results indicated discrepancies between the original and re-abstracted records. The vast majority of these discrepancies occurred where hospitals typed a diagnosis as Type 1 or Type 2, and the re-abstractor either did not record the diagnosis, or assigned to it a Type 3. Conditions listed on CIHI's Complexity Grade list that are Typed as a 1 or a 2 affect RIW and ELOS assignment, whereas Type 3 diagnoses do not. There were also discrepancies between the original and re-abstracted Most Responsible Diagnoses (MRDx). As a result of these discrepancies, there was a shift in CMG assignment, which resulted in changes in both RIW and ELOS values.

The Audit demonstrated that the coding of co-morbid diagnoses that had no significant impact on patient length of stay and/or treatment was systematic at some hospitals. This practice had a large impact on RIW and ELOS values.

To achieve the objective of helping the MoHLTC understand the observed variation in hospital case mix, the Audit identified the major causes of the variation and quantified their respective impacts on RIW and ELOS values. Also presented are a series of recommendations on how to best improve the quality of Ontario's acute care clinical data. The recommendations suggest a broad course of action that affect the MoHLTC, CIHI, CHIMA, OHIMA, Ontario hospitals, and the College of Physicians and Surgeons of Ontario. It is clear that the active support of each of these stakeholder groups is essential if the quality of these data is to be improved, and the data are to be applied effectively to planning, evaluative, and funding activities.

Results

This section presents the results for discrepancies from the 1,398 re-abstracted records and their impact on RIW and ELOS values at the hospital and CMG level.

Hospital Specific and Hospital Type Group Results

A total of 4,450 discrepancies were identified between the original and re-abstracted data. Forty-eight percent (2,125) of these discrepancies occurred because the re-abstractor found that diagnoses on the original abstract had no significant impact on the patient's treatment or length of stay. Another 15.5 percent of the discrepancies occurred where the re-abstractor found that the original record had been coded contrary to CIHI's guidelines.

The total RIW value for all 10 hospitals decreased by 17.5 percent following re-abstraction, and the total ELOS value decreased by 18.9 percent. There was a wide range in the percentage change in RIW and ELOS values across the 10 hospitals; the maximum change in RIW value was a decrease of 33.4 percent, and the minimum change was an increase of 0.8 percent. The differences in variation between the Teaching hospital group and the Community hospital group were more moderate. The table below presents the changes in RIW and ELOS values by hospital, hospital type group, and for the 10 hospitals together.

It is important to note that even for the hospitals that demonstrated only small changes in RIW and ELOS values, many discrepancies were still found. At the hospital with the smallest percentage change in RIW value, 140 records were re-abstracted and 420 diagnoses discrepancies were found; 193 of these were the result of diagnoses that had no significant impact on a patient's treatment or length of stay.

Within a CMG group, additional Type 1 or Type 2 diagnoses that are found on the Complexity Grade list can increase the complexity level (Plx(TM)) assigned to a record. For the selected CMG groups, there are four possible Plx levels:

• Plx Level 1 - No Complexity
• Plx Level 2 - Chronic Condition
• Plx Level 3 - Serious Important Conditions
• Plx Level 4 - Life Threatening Conditions

One of the factors that increases the RIW and ELOS values assigned to a record is a higher Plx level. For all the hospitals together, there was a 107 percent increase in the number of Plx level 1 cases following re-abstraction. The percentage increase in Plx level 1 cases ranged across hospitals from a low of 41 percent to a maximum of 159 percent. As it is not meaningful to compare Plx levels from different CMGs, these values do not include records where the CMG assignment changed following re-abstraction.

Case Mix Group Level Results

To investigate differences in the extent of variation at the individual CMG level, the audit results are also aggregated by CMG group. The total RIW and ELOS values decreased for all CMG groups following re-abstraction. There was a smaller range in the magnitude of variation across CMG groups than there was across hospitals. Due to discrepancies in the identification of MRDx, many records switched CMG and MCC groups. The table below presents the change in RIW value and ELOS, and the percentage of records that switched CMG and MCC groups for each of the selected CMG groups.

In total, 284 records (20.3 percent) switched CMG group following re-abstraction, and 194 (13.9 percent) switched MCC group. CMG 485 - Nutritional and Miscellaneous Metabolic Disorders and CMG 751 - Septicemia were the two CMG groups most prone to switching CMG and MCC groups following re-abstraction. The switching of CMG groups also had a large impact on RIW values and ELOS. For the 284 records that were grouped to a different CMG group, the total RIW value decreased by 21.1 percent.

Causes of Variation

The Audit identified the following two factors as the most significant causes of the variation in hospital case mix:

1. Co-morbid Diagnosis Typing Discrepancies
   • From the 1,398 re-abstracted records, 3,028 diagnoses were inappropriately deemed significant to the patient's length of stay and/or treatment by the original abstractor. Of these discrepancies, 53 percent (1,596) were re-abstracted as secondary diagnoses (of no significance to the patient's length of stay or treatment), and in 47 percent (1,432), the re-abstractor did not include the diagnosis at all in the patient's record.
   • In 56 instances, the reverse situation occurred where the diagnosis was typed as a secondary diagnosis, while the re-abstractor typed it as a co-morbid condition.
2. Misidentification of the Most Responsible Diagnosis
   • Out of the 1,398 re-abstracted records, 307 MRDx discrepancies were found. This represents a discrepancy rate of 22 percent. One hundred and sixty of these discrepancies occurred when the diagnosis identified by the re-abstractor as the MRDx had been originally coded as any other type.
   • Sixty-five (21 percent) of the MRDx discrepancies occurred where the re-abstractor did not include the diagnosis code identified in the original data as the MRDx anywhere in the record. In 55 (18 percent) of the MRDx discrepancies, the diagnosis code identified by the re-abstractor as the MRDx was not included in the original record.
   • The discrepancies for the diagnoses identified as the MRDx discrepancies caused 194 (14 percent) records to be assigned to a different MCC group after re-abstraction, and for 284 records (20 percent) to be assigned to a different CMG group. Where records switched MCC group, there was a 31 percent decrease in RIW value, and where records switched CMG group, there was a 21 percent decrease in RIW value.

Conclusions

There were many important discrepancies found between the original and re-abstracted records. These discrepancies had large impacts on both RIW and ELOS values, and also affected the grouping of cases into appropriate CMG and MCC groups.

The majority of discrepancies occurred where the re-abstractor disagreed with the original abstractor on the assessment of the clinical importance of diagnoses. This was the underlying cause of the following:

• 307 (22 percent discrepancy rate) discrepancies in the identification of the Most Responsible Diagnosis
• 3,028 (56 percent discrepancy rate) co-morbid diagnosis typing discrepancies.

CIHI has specific guidelines addressing how the Most Responsible Diagnosis ought to be selected and how co-morbid diagnoses should be typed. Given the large number of both of these types of discrepancies uncovered during the Audit, it is clear that these guidelines were not well adopted by hospitals, and/or the guidelines were too open to interpretation.

CIHI also has specific guidelines on how and when the results of diagnostic tests should be used as determining factors in the selection of the MRDx and the typing of co-morbid conditions as some diagnostic tests are used to increase the specificity of a code. Diagnoses like Septicemia (CMG 751) and Pneumonia (CMG 143) are highly dependent on the results of diagnostic tests. Given the high degree of variation observed in both these CMG groups, it may be that these guidelines are not well adopted by hospitals, and/or they are too subject to interpretation. Another source of variation that is difficult to quantify is the completeness of the patient chart at the time of abstraction. In some instances, the re-abstractor may have been using a more completed chart than was available to the original coder. The re-abstractor would therefore have had less need for interpretation.

Given the large decreases in RIW and ELOS values observed following re-abstraction, there was no evidence found to suggest that the variation in hospital case mix was caused by an increase in the acuity of patients cared for in the audited hospitals.

Because this Audit investigated only five medical CMG groups, the results presented here may not be representative of the entire DAD. However, the results of this study raise questions as to whether one would find a similarly high frequency of discrepancies should an audit of other complex medical CMG groups be performed. Indeed, other re-abstraction studies of Ontario's DAD have shown smaller but similar rates of discrepancies.¹ These studies were based on a random sampling of records for clinical conditions associated with specific health indicators, (including surgical CMG groups where it is assumed less opportunity for variation exists) as opposed to the methodology used for this Audit where records were randomly selected from only five CMG groups.

The frequency of discrepancies found between the data originally submitted to CIHI and the reabstracted data is salient. Given the important and ever-expanding application of these data to planning, evaluative, and funding purposes, it is clear that a broad and rigorous data quality strategy should be developed and implemented in Ontario. Below are a series of recommendations that should to be included as part of the data quality strategy.

Recommendations

Based on a detailed review of the findings of this Audit, the CHIMA and OHIMA propose the following recommendations to increase the quality of Ontario's Discharge Abstract Data set:

1. Standardization of Coding
   • The MoHLTC should mandate that all health information management professionals attend CIHI's Diagnosis Typing/Standards Workshop.
   • As part of its grouper redevelopment efforts, CIHI ought to seek out wide stakeholder collaboration on the refinement and redevelopment of Grade List Diagnoses. CIHI should consider the development of CMG-specific grade lists.
   • CIHI should also solicit increased consultation regarding the definition of diagnosis typing. The findings of this consultative process should inform any potential redevelopment of the diagnosis type definitions and standards.
2. Improve Clinical Documentation
   • The MoHLTC should review the regulations relating to the content of the health record from the Public Hospitals Act and ensure that all relevant regulations are being implemented.
   • The College of Physicians and Surgeons of Ontario should lobby medical schools to incorporate teaching the importance of high quality and timely clinical documentation in their medical student training curriculum.
   • CIHI ought to ensure that their coding standards allow for the capture of diagnoses based on evidence/documentation from all members of the clinical team involved with a patient's care.
3. Understanding the Necessity of Data Quality Accountability
   • The MoHLTC should lead the establishment of a broad data quality strategy involving wide stakeholder representation. This strategy should include a regular data quality monitoring and measurement process, including continued re-abstraction studies.
   • The MoHLTC should investigate including provisions for data quality in Hospital Service Agreements.
   • CIHI's redevelopment of the CMG grouper should take into account the findings of this study.
   • Hospitals should ensure that every staff involved in coding health records is a certified health information management professional, which includes CHIMA membership renewal on a yearly basis.
   • Hospitals should enforce Continued Professional Education workshops for coding and data quality standards.
   • Hospitals ought to be held accountable to the MoHLTC for the quality of the data they submit to CIHI.

These recommendations were designed to involve, and assign accountabilities to all stakeholder groups that have the ability to improve the quality of Ontario's acute care abstracted data. To successfully improve the quality of these data, each of the identified groups will have to collaborate closely with the others. A strong commitment from all will also be necessary. As the stakeholder most able to enforce rules and regulations, the MoHLTC should assume the lead responsibility for the development and implementation of a robust provincial data quality strategy.

Endnote

1. A) Discharge Abstract Database--Data Quality Study, CIHI 2002.
   B) Case Mix Group (CMG/Plx)--Data Quality Study, September 2002.
   C) Report of the Ontario Data Quality Re-abstraction Study, April 1991.