The Quality Collapse: The Turning Point

The Seven Pillars of Data Management, a novel in seven acts.

Chapter II, The Quality Collapse. Pillar II: Data Quality. Part 4 of 5.

The Reckoning

The regulatory and legal consequences arrived in waves, each one larger than the last, like a storm surge advancing up a shoreline that had been built too close to the water.

The College of Physicians and Surgeons of Ontario opened an investigation into the prescribing practices of every physician who had accessed a record later identified as corrupted. The Ontario Ministry of Health initiated a review of Lakeview’s CareConnect implementation under the authority of the Excellent Care for All Act. The Information and Privacy Commissioner of Ontario launched an investigation under the Personal Health Information Protection Act, PHIPA, which governs the collection, use, and disclosure of personal health information in the province and imposes specific obligations on health information custodians to ensure the accuracy of the records they maintain. Quebec’s Commission d’accès à l’information opened a parallel investigation for the Lakeview facilities serving patients under Quebec health cards, invoking the Act respecting the protection of personal information in the private sector and the newly strengthened provisions of Law 25. Health Canada’s Canada Vigilance Program requested a review of adverse drug reaction reports from Lakeview facilities going back to 2019.

And then the class action was filed.

Beaulieu v. Lakeview Health Alliance was filed in the Ontario Superior Court of Justice on November 18th. The statement of claim was forty-seven pages long. It named Lakeview Health Alliance, its board of directors, its Chief Information Officer, and the consulting firm that had performed the data migration as defendants. It alleged negligence, breach of fiduciary duty, breach of the duty of care owed to patients under the Regulated Health Professions Act, violations of PHIPA’s accuracy requirements, and, in the paragraphs that made the front page of the Ottawa Citizen and the Globe and Mail and the national broadcast of CBC’s The National, medical malpractice arising from systemic data quality failures that the defendants knew or ought to have known about and failed to remediate.

The proposed class included every patient whose record had been affected by the 2019 data migration. 412,000 people. The damages sought were $1.2 billion.

---

The Diagnosis

Evelyn Chow did not wait for the regulators to tell her what was wrong. She knew what was wrong. She had known, in some inchoate way, since her first month as Chief Medical Information Officer, when she had asked to see the data quality audit reports from the CareConnect migration and been told that no comprehensive audit had been conducted because the project had been delivered on time and on budget and the steering committee had signed off and the system was “live and operational.”

Live and operational. As though “live” meant “correct.” As though “operational” meant “trustworthy.” These were the words of an organization that had confused the deployment of a system with the quality of the data inside it: a confusion so pervasive in healthcare IT, in financial services IT, in every industry that has ever undergone a digital transformation, that it has become almost invisible. The system works. The reports run. The dashboards are green. And underneath the surface, the data is rotting.

Evelyn spent two weeks writing a document that she titled, with deliberate plainness, “Data Quality Assessment and Remediation Plan.” It was not a document written for regulators or lawyers or the media. It was a document written for the clinicians and administrators who would have to fix what was broken, a clinical diagnosis of an institutional disease, expressed in the language of the seven steps that she believed were necessary for Lakeview to survive.

Step One: Define Quality Dimensions. Lakeview had never defined what “quality” meant for its data. There were no standards for accuracy, no thresholds for completeness, no metrics for consistency. The CareConnect system had a field called “Allergies.” But nobody had defined the rules for how that field should be populated, validated, or maintained. Nobody had specified what “No Known Allergies” meant, did it mean the patient had been asked and reported no allergies? Did it mean the field had been reviewed by a clinician? Or did it mean, as it did in Thomas Beaulieu’s case, that the field was simply empty, and the system had displayed a default value that was clinically indistinguishable from a verified negative?

Evelyn established six dimensions of data quality for every clinical field in CareConnect: accuracy (does the data correctly represent the patient?), completeness (are all required fields populated?), consistency (does the same patient have the same data across all points of care?), timeliness (is the data current?), validity (does the data conform to defined formats and business rules?), and uniqueness (is each patient represented once and only once?). Each dimension was given a measurable threshold, a monitoring mechanism, and a responsible owner.

Step Two: Profile the Data. Before Evelyn could fix the data, she needed to understand its actual condition: not the condition that the integration project had assumed, not the condition that the steering committee had been told about, but the condition as measured by automated profiling tools applied to every record in the system. She deployed a data profiling platform across the entire CareConnect database and ran it for seventy-two hours. The results were the informatics equivalent of an MRI that reveals a tumour: 6.1 percent of all patient records contained at least one field that failed at least one quality dimension. In a database of 6.8 million records, that was 414,800 compromised files. The allergy registry was the worst: 11.3 percent of allergy records failed the accuracy dimension, meaning that one in nine patients in the system had an allergy profile that could not be verified as correct.

Step Three: Establish Data Quality Rules. Evelyn worked with the clinical leads of each department to codify rules for every safety-critical field in CareConnect. The allergy field received the most rigorous treatment: no allergy record could display “No Known Allergies” unless a clinician had explicitly verified and signed the entry within the preceding twenty-four months. Records that had not been verified would display “Allergy Status: Unverified” in red text: a visual flag that forced the clinician to stop, ask the patient, and update the record before prescribing. The distinction was critical: “No Known Allergies” was now a clinical assertion backed by a human being’s judgment. “Unverified” was an honest admission of uncertainty. The difference between those two states was the difference between Thomas Beaulieu going home with his family and Thomas Beaulieu going to the ICU.

Step Four: Implement Cleansing Processes. The remediation of 414,800 compromised records was the largest data cleansing operation in Canadian healthcare history. Evelyn designed it in three tiers. Tier One: automated correction of formatting errors, standardization of drug names and allergy codes to the WHO International Classification, and deduplication of records where the merge algorithm had created phantom patients. This addressed approximately 60 percent of the deficiencies. Tier Two: pharmacist-led review of every record where the allergy field had been flagged as potentially corrupted: 38,000 records, each one reviewed by a registered pharmacist against provincial drug information databases, original paper charts where available, and direct patient contact where necessary. Tier Three: physician-led clinical review of the 7,200 records where allergy data was actively wrong: records where, like Thomas’s, a false negative could result in a life-threatening prescription. Each Tier Three review required a face-to-face patient encounter, a fresh allergy assessment, and a signed attestation in the record.

The process took eleven months. It required 340 pharmacist-hours per week at its peak, supplemented by contract pharmacists from agencies in Ottawa and Montreal. The cost was $14.3 million, twice the budget that Gerald Foss had requested in 2014 to prevent exactly this scenario.

Step Five: Create Data Quality Dashboards. Evelyn built a real-time data quality dashboard that was displayed on monitors in every nurses’ station, every pharmacy, and every physician’s lounge in the Lakeview network. The dashboard showed five numbers: the percentage of patient records meeting all six quality dimensions, the number of unverified allergy records in the facility, the average time to resolve a data quality alert, the number of near-miss events attributed to data quality issues in the current month, and the facility’s overall data quality score on a 100-point scale.

The dashboards were controversial. Some physicians found them intrusive. Some administrators worried they would erode confidence in the system. Evelyn’s response was unyielding: “The alternative to transparency about our data quality is the illusion that our data is trustworthy. That illusion nearly killed a child. I will not participate in maintaining it.”

Step Six: Assign Root Cause Ownership. Every data quality deficiency identified during the profiling was traced to its root cause and assigned to a responsible party for remediation. This was not a popular process. Root cause analysis, when applied rigorously, has a way of eliminating the comfortable ambiguity that allows institutions to treat failures as “systemic”: a word that, in Evelyn’s experience, was often used to mean “nobody’s fault.” She rejected the word. Every deficiency had a cause. Every cause had an owner. The migration team’s merge-and-monitor policy was owned by the project manager who approved it. The underfunding of the validation program was owned by the steering committee that rejected Gerald Foss’s request. The absence of data quality standards was owned by every clinical and administrative leader who had been present during the integration and had not demanded them.

The root cause analysis produced a report that was 127 pages long and that read, Evelyn would later say, like the autopsy of an institution’s conscience.

Step Seven: Monitor Continuously. The final step was the creation of a permanent data quality monitoring program: not a project, not an initiative, not a twelve-month remediation plan with a start date and an end date, but a continuous, embedded, non-negotiable function of the organization. Evelyn staffed a Data Quality Office with six full-time health informaticists, a clinical pharmacist, and a biostatistician. The office reported directly to the CEO, not to IT, not to finance, not to any function that might be tempted to trade data quality for operational convenience. Automated monitoring scanned the entire CareConnect database every twenty-four hours, flagging any record that fell below quality thresholds. Monthly quality reports were presented to the board of directors. Quarterly audits were conducted by an independent health informatics firm.

Data quality at Lakeview was no longer an afterthought bolted onto the side of a technology project. It was a vital sign, monitored with the same vigilance that a nurse monitors a patient’s blood pressure or oxygen saturation, because in healthcare, data quality is not an administrative concern. It is a clinical one. When the data is wrong, patients bleed.

← All writing