MADiE, an eCQM Authoring & Testing Tool: Prototype / Usability Testing
Background
MADiE (Measure Authoring Development Integrated Environment) is a tool that enables Measure Developers to create and test electronic clinical quality measures (eCQMs). This study describes a prototype testing project conducted by Chris.
eCQMS
Electronic clinical quality measures (eCQMs) provide a standardized digital format to evaluate healthcare quality using data from electronic health records (EHRs) and health IT systems. eCQMs enable rapid assessment of healthcare standards and support government efforts to promote data interoperability in the healthcare sector.
MADiE’s Key Functionality
MADiE provides functionality that facilitates the authoring and testing of eCQMs with features by allowing Measure Developers to:
Programmatically Define eCQMs: Utilizing Clinical Quality Language (CQL), Measure Developers code an eCQM.
Test eCQMs by Facilitating the Creation of “Test Cases”: Test Cases verify eCQM performance by measuring results against simulated patient scenarios / clinical data.
Add Supporting eCQM Metadata and Documentation: Contextual information about an eCQM, such as its purpose, methodology, and relevant clinical guidelines provides critical details in understanding it and interpreting eCQM results.
Critical Context: Testing eCQMs by Simulating Clinical Scenarios
Creating Test Cases
When creating test cases (i.e. validating whether an eCQM is functioning as intended), Measure Developers programmatically create “mock patients”, often specifying key details about each such as:
Age
Gender
Health conditions
Healthcare services rendered (or not rendered)
Timing around healthcare services
etc.
Referencing eCQM Code & Validating an eCQM
To aid coding of test cases, developers reference an eCQM's underlying code/CQL. An eCQM’s code indicates how test cases will be calculated and interpreted. Measure Developers can then confirm which patient scenarios should—and shouldn't—produce certain results and construct test cases accordingly.
Once test cases have been constructed, they can verify whether an eCQM is working as intended (or not intended). If test cases produce the expected results, it confirms the eCQM is functioning correctly but if not, it can indicate that an eCQM has been constructed incorrectly.
Example
Let's say an eCQM targets patients 18 years or older. To verify its accuracy, a Measure Developer might create a hypothetical patient with a birthdate at least 18 years in the past in MADiE. Next, the measure developer engages MADiE functionality indicating that the test case will produce a "Pass" result, as the patient should meet the eCQM's criteria.
MADiE's "run test" function calculates the result. A "Pass" outcome confirms both the test case design and the eCQM's underlying logic.
However, a "Fail" result would trigger an investigation. First, the Measure Developer double-checks the test case for errors, like an incorrect birthdate. After fixing any issues, they re-run the test, hoping for a "Pass."
If the "Fail" persists and the developer is certain the test is correct, the focus shifts to potential errors in the eCQM's code/CQL. Perhaps it's mistakenly set for patients 19 or older, not 18. Once the error is fixed and the test re-run, a "Pass" result should validate the correction.
The Problem
The issues in focus for this study specifically relate to referencing an eCQM’s code/CQL while building a test case (as mentioned above). As described earlier, referencing the code/CQL helps Measure Developers determine which patient scenarios should (or shouldn't) lead to passing and failing eCQM's calculations. While referencing, Measure Developers:
Locate Code: Measure Developers must find relevant code snippets that describe the calculation of various patient attributes (e.g. age, gender, procedures undergone, etc.).
Build Test Cases: Cases must precisely illustrate the scope and limitations of the code's logic, and more broadly, of the underlying eCQM.
Challenges uncovered include:
Vastness: eCQM code/CQL often exceeds 200 lines, making it overwhelming to navigate.
Scattered Information: Relevant code snippets are dispersed throughout the code, requiring frequent navigation between dispersed code.
Poor Orientation: The narrow display lacks visual cues. After navigating more than 10-20 lines of code, it's difficult to identify which part of the code is being displayed. The narrow display spanned just 1/3 of the display in a non-wide, computer monitor (5:4 aspect ratio).
The Prototype
Several design improvements were introduced addressing the challenges above including:
Wider Display: The code/CQL display was redesigned to better utilize the entire screen, allowing Measure Developers to see more code/CQL, have more of a bird’s-eye view, and allowing them to more easily recognize which part of the code/CQL is displayed. That was accomplished in two ways.
Full Monitor Width: The area of the UI where the code/CQL was displayed was redesigned to use the full width allowed by a monitor, allowing it to span from edge to edge. The UI was originally designed for older generation monitors with a 5:4 aspect ratio, so when displayed on a wide screen monitor many use today, the UI remained in the center of the screen, flanked on both sides with empty space.
Increased Visibility: The UI area where the code/CQL was displayed was further redesigned, taking some screen area away from test case creation/configuration functionality and redistributing it to the code/CQL display area. Originally, the shared area was distributed at a 2-to-1 ratio, test case creation/configuration on the left side, taking the larger share and the code/CQL display on the right side, taking the smaller.
Quick Links
Direct Access to Key Structures: In previous research, it was learned that Measure Developers always navigate to the same fundamental code/CQL structures (e.g. Initial Population, Denominator, Numerator, Definitions, Functions, etc), present in almost all published eCQMs. One-click links allow instant navigation to essential code sections (Initial Population, Denominator, etc.), mirroring common developer search patterns.
Streamlined Navigation: This eliminates the need for repeated "Ctrl+F" searches, saving time and effort.
Prototype Testing Goals
This prototype evaluation, part of a larger testing effort, aimed to:
Assess Navigation Improvements: Determine if the redesigned interface enhances users' ability to reference and navigate within eCQM code/CQL.
Evaluate Quick Link Usability: Measure how successfully participants interact with quick links for streamlined navigation.
Identify Usability Issues: Identify any remaining usability challenges within the prototype.
Participant Recruitment
Recruitment for this study leveraged participants from previous generative research uncovering the issues that this prototype testing project addresses. Screening criteria for this study required participants to have experience with and be proficient in:
Utilizing Bonnie (MADiE’s predecessor) to author and test eCQMs.
Creating individual test cases
Referencing eCQM code/CQL to build test cases
Engaging with Bonnie’s equivalent of MADiE’s “run test” functionality
Fielding Research
To prepare for research day, Chris collaborated with the UX Designer and Product Owners to design a discussion format, draft a script, and create a semi-interactive prototype for MADiE.
Chris conducted 8 video-recorded, remote research sessions to test the prototype. Sessions focused on user needs related to eCQM testing and simulated the new code/CQL experience, with participants creating test cases in the redesigned UI.
Data Analysis & Reporting
Video recordings of the prototype testing sessions were analyzed using Dovetail. That allowed Chris to extract key insights, highlight important moments, and tag relevant data. Chris compiled these findings into a structured report and presented them to the team, along with any applicable recommendations.
Key Findings: Fast & Easy Navigation
All study participants reported a vastly improved navigation experience when referencing eCQM code/CQL within the redesigned prototype.
Key Findings: Further Navigation Enhancements
Participants provided valuable feedback to further enhance the navigation experience within the new prototype. Key suggestions included:
Inline Code Snippets: Displaying referenced code snippets directly alongside the parent code for improved context and reduced search effort.
Remove Jumping and Just Display Critical Code Structures: Second, instead of having “quick links” “jump” to various fundamental code/CQL structures within an eCQM’s code/CQL, another participants suggested that each “quick link” simply display the underlying fundamental code/CQL structure. Navigation in a sense would be eliminated, as each “quick link” would have an independent display showing its corresponding fundamental code/CQL structure.
Adjustable Display: Allowing users to customize the divider between the test case creation area and code/CQL display for optimized workspace flexibility.
Data Analysis & Reporting
Video recordings of the prototype testing sessions were analyzed using Dovetail. That allowed Chris to extract key insights, highlight important moments, and tag relevant data. I compiled these findings into a structured report and presented them to the team, along with any applicable recommendations.
Key Findings: Fast & Easy Navigation
All study participants reported a vastly improved navigation experience when referencing eCQM code/CQL within the redesigned prototype.
Key Findings: Further Navigation Enhancements
Participants provided valuable feedback to further enhance the navigation experience within the new prototype. Key suggestions included:
Display of Referenced Code Snippets: Some code contained references to other code snippets requiring further navigation to fully examine the initial code. Displaying referenced code snippets alongside the initial code would reduce the overall search effort.
Remove Jumping and Just Display Critical Code Structures: Second, instead of having “quick links” “jump” to various fundamental code/CQL structures within an eCQM’s code/CQL, another participants suggested that each “quick link” simply display the underlying fundamental code/CQL structure. Navigation in a sense would be eliminated, as each “quick link” would have an independent display showing its corresponding fundamental code/CQL structure.
Adjustable Display: Allowing users to customize the divider between the test case creation area and code/CQL display for optimized workspace flexibility.
Next Steps
Following this study, the team further enhanced the designs, incorporating the new feedback and further optimized the eCQM testing experience.