What the FCC can teach other feds about moving to the cloud

This is what remains after the FCC shifted 200 servers to a commerical data center. (Greg Otto)

The on-premise data centers at the Federal Communication Commission’s headquarters are a mess. Wires are hanging out of the floor, discarded monitors are strewn about and server racks are all out of place.

FCC chief information officer David Bray loves this mess. It means the agency’s data center is empty and the move to the cloud was successful.

There was a time over this past Labor Day weekend where it looked like that move could have been a failure. During the process of moving 200 servers and 60 racks to a commercial data center in West Virginia, Bray’s team of agency IT staff and contract workers rescued the project after spending 55 consecutive hours replacing all the cabling necessary to turn the system back on. What could have sunk the entire weekend project was reduced to a speed bump: After an additional three-day delay, the FCC’s systems were fully back on online by the end of the following week.

A failure would only have fueled the rising political firestorm surrounding the FCC — the result of the partisan controversy surrounding net neutrality and the looming leadership change that could come with the approaching election. However, Bray believes fears about political repercussions have left federal IT in its current state, where legacy systems persist because the political climate is so contentious that failure on any high stakes project could kill a decades-long career in an instant.

“They see [modernization] as ‘I only have a year, or two years, before the political appointee changes, so why do I want to risk my government career on something that may not work out?’,” he said. “That’s how you get into the situation where you have systems that are 18, 19 years old.”

Bray gave credit to FCC Chairman Tom Wheeler and Managing Director Jon Wilkins for buying into the need to update systems that hadn’t been updated since the 90s, instead of passing through, like their predecessors, without any real updates.

“They understood that it was going to be risky and that things weren’t going to go according to plan, but we were going to get through it and that helped,” Bray said.

“They also understood we had to do this now. I’m sure when Chairman Wheeler arrived he would have preferred to have everything pristine and new, but no one else had done it prior to him and each person had kicked it down the road. So if he didn’t do it, who would?”

Now, six weeks after the shift, Bray and FCC Deputy CIO for Resiliency Christine Calvosa spoke with FedScoop about the lessons learned and the road ahead as the agency moves forward.

Contingency and Communication

It seems obvious, but Bray and Calvosa could not place enough emphasis on the planning that went into the shift weeks before hand.

Calvosa said the weekly meetings, led by FCC’s project manager, Kevin Jordan, were vital to putting everyone on the same page. The team constantly went over scheduling, risks and other issues until everyone was comfortable.

“All parties need to be part of the process from the beginning,” Calvosa said. “Not just the operating team or the vendor who is hosting, but the teams who are testing our applications, who are ready to take support calls. Everybody needs to be aware of what’s happening, when it’s happening and be ready to support when phone calls start coming in.”

For this shift, the communication ended up being global. Bray said an IBM executive upended his vacation in India when he was briefed on the cable mishap, flying back to the data center to help remedy the problem.

“Everyone was in the trenches together,” Bray said. “That’s important because when things go wrong — and things will go wrong on something of this size — it could have resulted in finger pointing and people could have lawyered up. [The vendors] didn’t, which was good, because that wouldn’t have solved anything.”

Instead, teams took round-the-clock shifts fixing the problem, sleeping on the data center floor until everything was back on track.

“I think we bought all the Red Bull in the county,” Bray said.

The team of FCC IT staffers and vendors that helped stand up the agency’s servers at a commerical data center in West Virginia. (FCC via YouTube)

The Force Multiplier

Now that everything is up and running, the FCC has more money and manpower to concentrate on modernizing FCC’s online services. While the one-time cost of the shift was around $2 million, Bray says the cost savings are already being redeployed into projects like a revamp of the FCC’s licensing application and the Electronic Comment Filing System.

This re-use of savings been a problem for agencies. A September report from the Government Accountability Office found that 22 agencies were either not fully implementing OMB plans to reinvest their IT savings or not keeping track of how those dollars were being spent.

Calvosa said during her time at other agencies, so much of their budget was spent in the operations and maintenance of legacy systems, there was never any time to focus on the future.

“In less than one year of being [at the FCC], we are pivoting that budget so that we use more in development and modernization than O&M,” she said.

“Eighty five percent of our resources were maintaining what we have,” Bray said. “Now, it’s around 50 percent.

No Small Feat

In the end, Bray says what many other government officials told FedScoop at AWS’ re:Invent last week: Standing up an enterprise in a hybrid cloud environment is hard, but it’s necessary for the government if they are to get out of managing legacy systems.

“It’s hard but it’s doable. Our goal is to go to public cloud,” he said. “What this gets us is the fungibility that we can retire servers, and as we do, we will see cost savings on the spot.”