‘Don’t let your AI strategy outpace your network strategy,’ say tech experts

As federal agencies expand their use of data-intensive AI agents and software, they must also ensure that their underlying network infrastructure can handle the unprecedented demands placed on their systems, a panel of government technology leaders warned.

Civilian agencies reported that more than 3,600 AI use cases were in progress this past year, a nearly 70% increase from the previous year, according to the Office of Management and Budget. However, if infrastructure planning lags behind AI software adoption, agencies risk critical workflow bottlenecks that can stall operations and jeopardize data security, executives from the Department of Commerce, Verizon, and Cisco cautioned.  

Without a comprehensive assessment of their network infrastructure needs, the shift from pilot projects to active deployment could overwhelm legacy systems and throttle operations, they said during a video panel hosted by FedScoop.

Traditional practices, such as acquiring higher-bandwidth pipes or backhauling traffic to centralized gateways, won’t necessarily meet these new demands.

Simply throwing raw capacity at the issue is a major misconception that fails to account for how internal routing and inspection points choke erratic data bursts, said Brian Epley, Chief Information Officer and Chief AI Officer for the U.S. Department of Commerce. Epley warned against underestimating the role of end-to-end transport in supporting AI.

“The pivot from generative AI to agentic AI… really requires that we bring compute and data together, and that transport is what’s critical,” Epley said. The transition demands highly adaptable environments that tightly integrate data sets with hardware accelerators. To get there, Epley said his office is focusing on software-defined agility, zero-trust alignment, dynamic bandwidth allocation and embedded micro-segmentation as part of a larger unification initiative called “One Commerce.”

The stakes are significant. The department comprises 13 independent bureaus, including NOAA and the Census Bureau, that rely on “massive environmental and scientific data sets that live across multiple cloud environments.” The goal, he said, is to “have continuous identity attribute verification…so that those autonomous agents continue to operate.”

Addressing network boundaries

Agency leaders must also recognize that designing AI-ready infrastructure requires “solving for two architectural realities” to scale AI without sacrificing security compliance, said Sam Lakey, Associate Director of Solutions Architecture at Verizon Business Group.

One involves addressing historical security frameworks, such as Trusted Internet Connections (TICs) and Managed Trusted Internet Protocol Service (MTIPS), that were created for a centralized processing world and required backhauling traffic through rigid access points.

“Real-time AI cannot wait for data to travel from the edge to a centralized MTIPS gateway and then back again. If you try and force modern, distributed AI through latency MTIPS, your AI initiatives can stall out on day one,” he said.

The second reality involves shifting “from physical choke points to a layered security approach, with telemetry-driven security. We need intelligent network fabrics that stream real-time telemetry to your CDM (Continuous Diagnostics and Mitigation) and security tools without killing throughput,” he said.

Cisco Sales Leader Justin Fields stressed that as agencies assess their requirements, they need to map infrastructure to data locations rather than distant centralized hubs to accurately right-size the environment.

“When you talk about AI, it’s 100% about the data,” Fields noted. “You have to connect the data. You have to secure the data. You have to visualize the data based on the agency needs… estimate how much throughput… and bandwidth is required, and how much data you’re going to be analyzing to help right-size the infrastructure.”

Failing to do so can lead to what Epley called “GPU starvation,” when an agency’s network can’t deliver data as fast as high-performance accelerators require and consume it. “That results in an expensive compute asset that may sit idle.”

Agencies need to think about “not just the components of the (AI) transaction” and the ‘tokenomics’ of training AI, but ensuring that the AI query costs are scaling predictably without surprises.”

“Agentic AI is an entirely new operational paradigm,” argued Lakey. “It doesn’t just consume data. It creates massive, erratic, unpredictable traffic patterns. If you build a model first and worry about the infrastructure later, your network could break.”

He recommended that, before agencies pilot a single AI tool, teams must collaboratively look at three functional areas:

• Data geography: Map the path of your data.
• Security:  Autonomous agent acts like a superuser. Design your zero trust and micro-segmentation around AI itself, treating it as an independent entity.
• Network elasticity: AI traffic is notorious for spiking. Your network must be intelligent, software-defined, and capable of scaling dynamically.

“Don’t let your AI strategy outpace your network strategy,” he concluded.

This article and the original video panel were produced by Scoop News Group for FedScoop and sponsored by Verizon and Cisco.

Learn more about how Verizon and Cisco are helping public sector agencies ensure their network infrastructure is AI-ready.