Kratsios: NIST needs ‘to go back to basics’ on standards for AI, not safety evaluation

The director of the White House’s science and technology arm indicated interest Thursday in seeing the National Institute of Standards and Technology focus more on standards for AI and less on model safety evaluation.

The Office of Science and Technology Policy’s Michael Kratsios told an audience gathered by the Consumer Technology Association — a member group representing technology companies — that the Biden administration “hijacked” NIST and “and turned it into a safety model evaluation agency.”

When the Biden administration created a safety institute at the standards agency and then used it to run “x-risk evals, I think we kind of lost our way there,” he said. (“X-risk” is a shortened term for “existential risk” that’s associated with the idea that AI poses major threats to humanity.)

“To me, I think we need to go back to basics at NIST, and back to basics around what NIST exists for, and that is to promulgate best-in-class standards and do critical metrology or measurement science around AI models,” Kratsios said. 

Once the agency accomplishes that and produces those standards, “then the industry can coalesce around it, and they can do all the evals that they ever imagined,” he said. 

Kratsios’s comments about the body once known as the AI Safety Institute came a day after the White House released its anticipated AI Action Plan — which made dozens of recommendations to do things like deregulate and rid AI of “ideological bias” — as well as three executive orders that set parts of that plan into motion. The Thursday panel, moderated by CTA’s CEO and vice chair Gary Shapiro, was focused on those actions. 

The discussion also followed the Trump administration’s move last month to rename the NIST-located safety institute to the Center for AI Standards and Innovation, cutting “safety” from the name. That component was initially announced by the Biden administration in November 2023 at the UK AI Safety Summit and, over the next year, focused on working with industry, establishing testing agreements with companies, and conducting evaluations.

After the renaming, NIST’s parent agency, the Department of Commerce, indicated that the component would continue with much of the same work, including efforts to evaluate AI technology that may threaten national security and assess U.S. systems and those of the country’s adversaries. The AI Action Plan also made several recommendations that included the AI standards component, such as building national security-related evaluations and partnering with the cyber industry to include AI in standards and frameworks. 

“We need to be able to be in a position where we’re all talking the same language around how you do an eval. What does a good eval look like?” Kratsios said, adding that what a good evaluation looks like will be dependent on what industry the model is targeted toward.

“I think all those sort of sector-specific evals will be possible if we’re able to actually do a much better job on defining what the standards are around evals more broadly,” Kratsios said.