The Department of Veterans Affairs has been working aggressively to eliminate dozens of instances of a so-called broker application that for the past decade has created an internal security vulnerability that could allow individuals with the right set of skills and tools to gain unauthorized access to veterans’ data, the VA confirmed in an exclusive interview with FedScoop.
Message brokers, which carry out remote procedure calls for machine-to-machine communications inside VA’s network, are a central part of VA’s service-oriented architecture. They help create effective system communications for a variety of user applications requiring access to the VA’s main electronic health record system, known as the Veterans Health Information Systems and Technology Architecture, or VistA. But as those user applications grew, so did the use of brokers. In 2003, one particular broker was developed in a way that did not pass the identity information of the application user to VistA, creating a situation involving internal anonymous access to VistA data.
A slide from an internal VA presentation on Managing the VistA Broker Feature, dated October 2014.“There is a broker that is in use at the VA that does that type of transaction, and it doesn’t send the identity of the individual who is doing the transaction. And that’s a problem for us because you’d like to know who is doing transactions,” VA Chief Information Officer Stephen Warren said in an exclusive interview with FedScoop. After 18 months of investigation and remediation work, VA has narrowed down the number of instances where this insecure broker software is in use to 21, according to Warren, who confirmed it is not available outside of the VA network. The VA told FedScoop that in three cases the insecure broker accessed the My HealtheVet portal, a bed management application and the suicide prevention hotline.
“Where we have a broker that’s not using the feature correctly, we’ve contained it, we’ve isolated it, we’re monitoring it and we’re also working it to very carefully replace it with a broker that has the appropriate features so that we can continue to give those life-saving services while also tightening up the risk that we have to the organization internally,” Warren said. “Folks were really excited because they had seen what [electronic data interchange] was doing and the ability to use Web services and they went out and deployed this broker with good intentions. So lots of folks were using it.”
Warren took the unusual step recently to issue a policy memorandum across the VA to stop the deployment of this particular broker software and require new brokers to be vetted as part of the VA’s many IT review processes. “I’ve gone out and said, ‘timeout folks … you’re not doing that anymore.’ We’ve built it into our design reviews, our architecture reviews … and all of the reviews that are taking place,” Warren said.
Warren has also directed his security team to identify the appropriate configuration management and settings on servers to ensure an insecure instance of the broker can’t simply be turned on by a developer inside VA’s network with authorized access. “They can’t just come in and install it or pull it out of a library because we’ve removed it from the library. We’ve taken it out of the library so you can’t build on it anymore,” Warren said. “We also put a lot of scanning in place … to make sure we have deeper and deeper monitoring, more comprehensive monitoring and overlapping technologies so that we are looking for instances, so we know what’s out there and if there’s a perturbation.”
A slide from the internal VA presentation shows some of the VistA applications that have been identified as using an insecure broker for remote procedure calls.In some instances, VA has actually been putting blocks in place that prevent deployment of the broker.
What started last year as a high-priority security project may still be two years from completion, Warren acknowledged. That includes all changes, controls that must be put in place and any redesign necessary on back-end systems to ensure “that this is a feature that can no longer be misused,” he said. “The longer-term piece is more than just working through the instances we have. It’s how do we change the architecture of VistA such that the only way you can use the broker is in a governed, service-oriented architecture framework using an enterprise bus that pretty much mandates how the feature is used.”
When asked if the internal broker vulnerability could have played a role in the recent nationwide scandal involving scheduling manipulation or if it presents a larger vulnerability to external hacking threats, Warren dismissed the possibility, arguing that the level of expertise required to exploit the broker vulnerability is well beyond the average VA user or scheduler.
“We know and we’ve validated you cannot do this type of transaction from outside the VA. You need to be an authenticated user that’s logged in,” Warren said. “We also know that to do this … you need to have pretty deep knowledge as well as some pretty good tools for you to be able to do this. You would need to be a developer, you would need to have some good knowledge and experience using RPCs in that VistA environment. You would also need to know an awful lot about the connection,” he said, including what the receiver of the data is, how the data is formatted, what the structure is and where you need to go for it.
“Nothing we have seen, nothing that has come from the independent reviews has shown that schedulers were able to tamper with the brokers,” Warren said. “Unfortunately, certain individuals were put in positions, and they didn’t do the right thing. This was not an IT failure and not a system failure. That cultural failure is being dealt with. That integrity failure is being dealt with.” Follow @DanielVerton
