What it takes to build accountable AI in government

Federal agencies are testing what AI can do, but the harder work is embedding it in the systems that run financial operations, grants and acquisition. Moving beyond standalone chatbots and copilots means connecting AI with the data, workflows and controls those functions already depend on.

That challenge is the focus of “Accountability in the AI Age,” a six-part video series produced by Scoop News Group for FedScoop and underwritten by SAP and Carahsoft. In it, federal officials from the State Department, the Interior Department, and the Export-Import Bank of the United States and industry experts from SAP and Carahsoft discuss how to turn AI pilots into everyday operational tools while keeping accountability for how government funds are managed.

Embed AI where the work happens

Tahera Zamanzada, Industry Executive Advisor at SAP, drew the core distinction: “The standalone AI pilots sit on top of your systems, and embedded AI runs through them.”

Agencies running established ERP systems aren’t behind, she noted, because years of transaction data, workflows and business rules give AI the context it needs. Her advice for leaders looking to transition from peripheral tools to the core: “You don’t have to boil the ocean, just start where the cost of getting it wrong is real.”

Caitlyn Lewis, Market Research Director at Carahsoft, said getting that value means looking beyond siloed platforms. “Government operations do not happen inside one system; they happen across the systems,” she said. The quickest wins emerge wherever skilled employees are “effectively acting as human middleware between the disconnected systems,” such as grants, permitting, case management and financial reconciliation.

From reactive to preventive

Connecting those systems lets agencies shift from auditing problems after the fact to preventing them in real time.

At the Interior Department, Andrea Brandon, Deputy Assistant Secretary for Budget, Finance, Grants and Acquisition, wants agentic AI that flags and stops improper payments before money leaves the building. Her current dashboards only surface problems after the fact, but as Brandon noted, “With a car, the engine light pops on right away. It’s not after you’re broken down on the road.” Because acquisition and grants systems feed financial and payment systems, “we need to see the whole spectrum.”

Maria Barcarse, Director of SAP’s Center of Excellence, offered a similar comparison: “External AI tells you you’re about to hit traffic, but embedded AI reroutes you while you’re still driving.” Agencies don’t need perfect data to get there, she said. They need data that is trusted, connected and acted on, because “a prediction that no one uses has zero value.”

Governance is the foundation

Strong governance lets agencies use AI with confidence. Howard Spira, SVP, CIO, and Chief AI Officer at the Export-Import Bank of the United States (EXIM), cautioned against treating technology as a quick fix. “AI is not a layer of magic dust. It’s an operating capability that has to sit on good process, good data and clear accountability.”

The bigger risk is harder to spot. “The danger is not that AI fails loudly. Actually, the danger is that it fails quietly,” he said. That’s why EXIM manages AI as part of its IT portfolio, subject to the same zero-trust, privacy and records controls as any other system.

The State Department takes a similar approach. Chief Data Scientist and Responsible AI Official Gio Altamirano Rayo said embedded AI tools go through the standard authorization-to-operate process: “We really do not take any shortcuts.”

What concerns her most is the “gradual erosion of clear human ownership over decision” as busy users defer to model output, along with model drift over time. Responsible AI, she said, isn’t a one-time exercise. “It’s an ongoing operational discipline.”

Leadership sets the pace

Several leaders said that technology is the easier part. The tougher challenges are organizational alignment and leadership mindset.

Barcarse predicted embedded AI will shift leaders “from managing processes to orchestrating decisions,” adding that it “doesn’t replace that leadership; it actually raises the bar for it.”

Zamanzada put it plainly: “That distinguishing factor is not a technology selection, it’s leadership courage, the willingness to commit to transformation rather than experimentation.”

Watch all six interviews on the “Accountability in the AI Age” website.

This video series was produced by Scoop News Group for FedScoop and sponsored by SAP and Carahsoft.