DOD’s $4B health record system pilots not suitable for use, report says
The Defense Department’s $4.3 billion MHS GENESIS electronic health record modernization project, which it hopes to roll out to 9.6 million beneficiaries by 2022, is “neither operationally effective nor operationally suitable” in its current state, concludes a new report on the initial internal testing and evaluation of the system’s first pilots in the Pacific Northwest.
The scathing report by the department’s Operational Test and Evaluation office details how the Cerner-based MHS GENESIS’s initial implementation at three military medical facilities in Washington state “does not demonstrate enough workable functionality to manage and document patient care” and has “poor system usability, insufficient training, and inadequate help desk support.”
“MHS GENESIS is not operationally effective because it does not contain enough functionality to manage and document patient care,” the report reads. “Users successfully performed only 56 percent of the 197 tasks used as Measures of Performance (MOPs). Non-standard data and the failure to adhere to Interface Control Documents (ICDs) hampered information exchange with interfacing systems.”
On the usability side, pilot users gave the system “an average score of only 37 out of 100 on the System Usability Scale (SUS), well below the threshold of 70 that indicates acceptable usability.”
It continues: “Training was insufficient to overcome usability problems, and a lack of documentation forced users to develop their own operational workarounds. User survey comments from the three [initial operational testing and evaluation] sites reported similar problems that included undocumented and inconsistent workarounds, excessive system latency, inaccurate patient information, badly assigned user roles, poor user training, uneven assistance from on-site trainers, and lack of visibility of the status of trouble tickets. Users from the four initial sites submitted 14,383 help desk tickets from January to November 2017, overwhelming the help desk’s ability to resolve them.”
Cybersecurity testing is still preliminary, the report explains, which leaves the system’s survivability in question. But the report did include an initial findings on the system’s detection and incident response, and data protection, both of which it found were inadequate, citing lags in response times and patient data that “is not protected in accordance with DOD standards.”
Finally, the report questions the system’s scalability, though no formal testing on the matter has been done, pointing to user-reported lag times and standardization issues across what will eventually be an enterprise system at more than 650 medical facilities.
After discovering the flaws at the first three pilot sites, the MHS GENSIS program management office decided to pause work at a fourth site “to remediate significant problems discovered at the first three sites.”
The Pentagon’s director of operational testing and evaluation recommends in the report that the team continue its pause until full testing is completed at the first three sites and all deficiencies are taken care of.
Experts familiar with the report told Politico, which first reported the story Friday, the findings are “devastating,” particularly the 156 “critical” or “severe” incident reports which could lead to the death of patients.
Defense officials, however, said progress has been made since the conclusion of testing in November. Last month, when pummeled with questions about the system’s delayed roll out, Stacy Cummings, leader of the Program Executive Office-Defense Healthcare Management Systems, told lawmakers her team won’t begin again implementing the system at new facilities until 2019. She still has faith, she said, the system will be fully operational by its 2022 deadline.
These findings also have the potential to impact the Department of Veterans Affairs, which decided last year to move to a Cerner-based EHR, like the DOD, so that the two departments’ systems would be interoperable.
