Law enforcement agencies need standardized AI field testing, presidential advisers say
A long-gestating recommendation on how federal law enforcement agencies should test the performance of artificial intelligence tools before implementing them is ready for the White House after an advisory panel to the president on AI approved the final findings Wednesday.
The National AI Advisory Committee unanimously voted in favor of the findings presented in the Law Enforcement Subcommittee’s 24-page document on the field testing of AI tools, a move that followed approval earlier this summer of three recommendations regarding the practice.
Jane Bambauer, the Law Enforcement Subcommittee’s chair, said during a public meeting Wednesday that the recommendations reflect the group’s findings on the subject — though it’s “still pretty early in this idea of testing AI in the field,” leaving the door open for the ideas to “be improved on.”
“This is one iteration,” Bambauer said of the recommendations and findings, “and by no means the sole means of understanding how an AI performs its function in the field.”
The subcommittee’s first recommendation was simply for the Office of Management and Budget to push federal law enforcement agencies to follow a checklist when testing AI tools in the field. A potential checklist compiled by the subcommittee includes callouts to describe the AI tool and its intended use, document use limitation plans, conduct an AI impact assessment, identify the testing method and complete a questionnaire designed to “brainstorm and identify metrics” associated with the system.
In an interview with FedScoop last month, Bambauer said the subcommittee did not want to give the impression that that checklist was “a static” or “final set of operations or protocols that should be followed.” Rather, the subcommittee intends for its checklist to serve as a “guidance document,” and the federal government should determine which “standardized approach” to AI field testing that law enforcement agencies must follow.
“The process of field testing should … provide transparency,” Bambauer said, adding that baseline values are imperative so that those administering the tests know “how much better or worse” the AI tool performs than its predecessors.
The subcommittee’s second recommendation calls on OMB to revise its guidance and require plans for AI field testing among federal law enforcement agencies to be made public, as well as the results from those tests — regardless of whether the tool is ultimately adopted.
The third recommendation, meanwhile, pushes for funding and research support for state and local law enforcement to conduct their own AI field testing.
The Law Enforcement Subcommittee first presented the initial version of its AI field testing recommendation to the full NAIAC body during a May meeting of the group, which comprises AI experts from the private sector, nonprofits and academia and operates under the banner of the National Institute of Standards & Technology.
For Bambauer, a University of Florida law professor with a quantitative research background, the thought process behind the recommendation was a long time coming.
Though President Joe Biden’s AI executive order “explicitly requires real-world testing” of AI tools, Bambauer said there wasn’t a “standardized approach to testing AI” in federal policing. The subcommittee decided that recommendations on AI field testing would fill an important gap.
Bambauer and her colleagues then went on a fact-finding mission with local law enforcement to find out how big a lift field testing might be. In a visit with the Miami Police Department — where subcommittee member Armando Aguilar serves as assistant chief — Bambauer said she expected to find a skeptical force concerned with too many testing protocols gumming up their work.
But the MPD, she said, conducted a study of AI tools in its real-time crime center and found that the process for testing “wasn’t onerous.”
“It was not a big deal to implement, and it actually gave them great confidence,” Bambauer said. “It’s a win-win … if we can really get the best information, high-quality information, [on] what’s working and what’s not.”
After Wednesday’s vote, the Law Enforcement Subcommittee’s recommendations and findings on AI field testing will be ready for submission to the president and the National AI Initiative Office. The subcommittee previously recommended that law enforcement agencies broaden their use case inventories and publish annual summary usage reports on facial recognition and other AI tools, among other suggestions.
