DOD reaffirms JEDI award to Microsoft after corrective action

After taking corrective action on part of the $10 billion Joint Enterprise Defense Infrastructure (JEDI) cloud contract, the Pentagon has reaffirmed its original award to Microsoft.

The Department of Defense said Friday it “completed its comprehensive re-evaluation of the JEDI Cloud proposals and determined that Microsoft’s proposal continues to represent the best value to the Government.” Effectively, the DOD has re-awarded JEDI to Microsoft.

Earlier this year, DOD decided to take corrective action on the JEDI acquisition and re-evaluated the proposals of Microsoft and losing bidder Amazon. Over the course of its ongoing claims court bid protest, Amazon made a strong case for at least one area in the contract where the department erred in evaluating bids. Rather than waiting for the court to decide on the matter, DOD asked for a remand of the protest to “reconsider certain aspects of the challenged agency decision.”

Despite DOD’s re-affirmation of the award, work with Microsoft is still on pause because of a court-ordered injunction. The department said it “is eager to begin delivering this capability to our men and women in uniform.”

A Microsoft spokesperson echoed the department’s sentiments: “We appreciate that after careful review, the DoD confirmed that we offered the right technology and the best value. We’re ready to get to work and make sure that those who serve our country have access to this much needed technology.”

Though DOD called its re-evaluation “comprehensive,” it’s unclear what all it took into account when looking at proposals again. Amazon’s original complaint brought up more than a handful of ways it alleges DOD erred, but the court so far only focused on one of those areas, which it said was probably enough of a misstep that Amazon would likely succeed in its protest.

The company pointed to this in a blog post Friday published after DOD’s announcement.

“Given the DoD did not agree to meaningfully review the many evaluation flaws outlined in our protest, we said the corrective action was likely to result in another contract award based on politics and improper influence and not based on the relative strengths of the two offerings,” the post says. “That is exactly where we find ourselves today, with the DoD’s re-evaluation nothing more than an attempt to validate a flawed, biased, and politically corrupted decision.”

Amazon also noted that its bid was much lower than Microsoft’s this time around “by several tens of millions of dollars.” In the original award to Microsoft last fall, the DOD cited price as a major factor in its decision, according to Amazon.

The company didn’t hold back in opining that it believes DOD’s attempt to take corrective action has been disingenuous and a waste of time: “‘Corrective action’ was used as a way to halt our litigation, delay further investigations and incorrectly give the appearance that only one issue needed to be fixed while giving the impression that the DoD was actually going to fix something. While corrective action can be used to efficiently resolve protests, in reality, this corrective action changed nothing, wasted five months that could have been spent getting to the bottom of these serious concerns, and was designed solely to distract from our broader concerns and reaffirm a decision that was corrupted by the President’s self-interest. When we opposed the DoD’s approach to corrective action, we predicted this would happen, and it has. By continuing to delay, distract, and avoid addressing these very serious issues, the DoD is turning out to be its own worst enemy with regard to speeding things along.”

Now, things go back to court, where the parties involved will discuss next steps and the judge will decide if DOD’s actions are enough to remove the temporary injunction on work.

CMMC accreditation board gets new members as several depart

The independent board tasked with implementing the new cybersecurity standards for defense contractors has some new faces after the recent departures of several members.

The Cybersecurity Maturity Model Certification Accreditation Body (CMMC-AB) added Sheryl Hanchar, vice president and chief information security officer of Cobham Advanced Electronic Solutions, and Charlie Williams, president of CWilliams LLC, to the board of directors. Both have military experience, with Hanchar serving in the Navy and Williams working as a civilian in Air Force contracting and director of the Defense Contract Management Agency. A role on the AB’s board involves much more day-to-day and tactical work, rather than the typical long-term strategy and advice usually associated with board seats.

“A regular rotation is expected and always brings fresh ideas and perspectives to our fast-growing organization,” Mark Berman, communications committee chair told FedScoop in an email.

The program the AB is implementing, CMMC, is the Defense Department’s new standards all contractors will need to comply with to earn government dollars. It replaces the current self-assessment standards with a third-party verification scheme where contractors must have networks that meet a one-five level of security controls. Where the AB comes in is they will train, accredit and oversee the ecosystem of assessors that will certify roughly 300,000 contractors. Most contracts will be a level one, which only involves basic security measures.

The two additions come after five members recently departed or stepped back from their work with the AB. One, Nicole Deans, has been on a personal leave of absence for several months, multiple people familiar with the matter have said. Other members who recently resigned include: Jim Gopel, Chris Golden, Valecia Maclin and John Weiler.

The turnover comes as the AB is close to executing a new contract with the DOD that would redefine the two entities’ relationship. DOD recently said the contract would be finalized by the end of August. A spokesperson did not return a request for comment on whether it’s been executed.

John Weiler continues to have a relationship with the AB through a memorandum of understanding that his nonprofit, the IT-Acquisition Advisory Council (IT-AAC), signed with the AB. Part of that work, according to Weiler’s council, has included building what it calls the “CMMC Center of Excellence.” The AB recently said that it does not have any intent to be a part of the CoE, nor does a copy of the MOU obtained by FedScoop mention a CoE by name. It does give IT-AAC authority to promote education and mutual interests with the AB.

“I resigned on my own volition, I was frustrated with a number of issues that I felt I could no longer impact,” Weiler told FedScoop about his resignation. He went on to say “I believed they would honor and support the MOU.”

The turnover is not unexpected for the AB. Members have regular virtual meetings, often stretching several hours long on minute details on the new regulations. Despite it being a volunteer position, it became a full-time job for many.

“I would still be involved but it started to become too time-intensive and was keeping me from doing revenue-producing work,” Gopel told FedScoop in a LinkedIn message. “It was an honor to be able to be part of such an immensely important project, and I hope to be able to contribute in other ways in the future.”

Army opens its new Cyber Command headquarters

Georgia is now officially the home of the Army Cyber Command headquarters, a move that consolidates soldiers and civilian hackers onto one base.

Fortitude Hall, located at Fort Gordon near Augusta, Georgia, is the new home for the Army’s hacking, cyberdefense and information warfare arm. The hall was dedicated Thursday after the Army’s years of planning to move its cyber specialists to a central headquarters.

Fort Gordon is also home to the Army Signals Corps, Cyber Center of Excellence and other network security-focused programs, which will allow the command to collaborate with mission partners.

Some of the 1,200 soldiers, civilians and contractors the Army anticipates hosting in the facility are already on-site, but the Army doesn’t expect it to be fully staffed until next September.

“We really have consolidated all our cyber capabilities in South Georgia,” James McPherson, the service’s No. 2 civilian, told reporters Thursday. “This is tremendous for all things cyber in the Army.”

The Army stood up its cyber command in 2010, and the construction of Fortitude Hall began a few years later in 2016. The building of the high-tech, highly secure facility experienced minor delays due to the coronavirus pandemic, according to the Army.

The Army, which has perennial problems with recruiting and retaining a cyber workforce, says that moving the command’s headquarters into a single campus will reduce the number of times service members and civilians will need to uproot their families and move to new locales.

“This gives us tremendous flexibility and connectivity,” Lt. Gen. Stephen Fogarty, Army Cyber Command’s commanding general, said Thursday.

By centralizing in southern Georgia, the command is in the state’s “cyber corridor,” where state and local officials have invested in cyber education and workforce programs. Senior Army leaders at the base said they plan on doing outreach to nearby schools, from elementary to college-level, to recruit future uniformed and civilian cyber specialists.

“Talent management is our No. 1 priority,” Fogarty said. “The weapon system doesn’t fight itself, so the workforce is the most important part of the equation.”

Having the command under one roof gives the Army a technical advantage as well. By physically locating its networks at Fort Gordon, instead of connecting to another location, the Army believes it has one of the strongest and most secure networks in the military.

“The IT is very powerful,” Fogarty said. He added that “we probably have the best network in the DOD right now within our facility.”

JAIC needs enhanced authorities to coordinate DOD AI efforts, Bob Work says

Former Deputy Secretary of Defense and artificial intelligence evangelist Bob Work says the DOD is at risk of losing the AI race and needs to rethink its departmentwide development model.

Work thinks that the Pentagon’s Joint AI Center needs stronger authorities over all of DOD’s projects to coordinate better between services and components. He said Wednesday at the AFCEA DC AI and ML Technology Summit that right now the military has a “one thousand flowers blooming” approach to its AI projects, with too many disparate ideas that won’t yield collective results fast enough. Instead, the JAIC should be a standards-setter and centralized architecture lead for AI across the DOD.

Within that enhanced JAIC, Work also suggested its scope could be broadened as well. He proposed the center add another “A” to its name, making it the Joint Automation and Artificial Intelligence Center, or JAAIC.

Work’s suggested strategy of centralizing requirements setting and decision-making reflects a historical precedent.

“I would recommend that we adopt a naval reactors-type model,” said Work, who is currently vice-chairman of the National Security Council on AI and the owner of TeamWork, LLC, .

The Navy harnessed nuclear power for its submarines and ships when Adm. Hyman Rickover took the reigns as “the Father of the Nuclear Navy.” The department was able to rapidly field the emerging source of power with the centralized zeal from Rickover. It’s a story Work says he wants to see with AI, giving the JAIC a Rickover-type authority.

His analogy gained some support, with the former head of the JAIC, retired Lt. Gen. Jack Shanahan, tweeting that the idea is “apt in some ways.”

“But that was [a] singular focus on one issue,” Shanahan added. “AI [is] much more like electricity or networks…an enabling capability.”

The application of AI in warfare Work most wants to see is AI-powered networks of information that can be paired with human operators to rapidly make battlefield decisions. It’s the core pillar of his “Third Offset Strategy,” a document Work authored during his tenure with the DOD. With a more “muscular” JAIC acting as both standards-setter and motivator for all DOD AI programs, Work thinks the nascent idea of Joint All-Domain Command and Control, the military’s network-of-networks concept, can become a reality much faster.

“We’re all saying the right things. AI is absolutely important. It’s going to give us an advantage on the battlefield for years to come,” Work said.

To amplify the power of the JAIC, the DOD needs a solidified public-private partnership as a means to compete with the Chinese civil-military fusion, Work added. The idea would be to create a bridge between the sectors, driving change for easy collaboration. In specific, Work said, the DOD should prioritize testing and evaluation with some setup akin to a national consortium for companies and academic institutions to help DOD test new algorithms.

“We may be losing the race due to our own lack of urgency,” he said

Oracle strikes out again in JEDI protest

Oracle has once again been dismissed in its quest to derail the Pentagon’s single-award $10 billion Joint Enterprise Defense Infrastructure (JEDI) cloud acquisition.

The company couldn’t convince a federal appeals court that the Department of Defense illegally structured the JEDI contract as a single-award procurement with “prejudicial,” competition-limiting gate requirements.

For several years now, Oracle has been trying to sue its way into contention for the JEDI contract, failing first to convince the Government Accountability Office and then the U.S. Court of Federal Claims that the DOD should open the contract to a multiple-award acquisition with more lenient gate requirements. After losing in federal claims court — which did find parts of the contract to be flawed, but not in any way that prejudiced Oracle — the company appealed.

The U.S. Court of Appeals for the Federal Circuit stood by the claims court’s decision, finding “no reversible error” and terminating the case this week.

It’s unclear what’s Oracle will do next — take the protest to the Supreme Court, cut its losses or return to claims court with a different approach. The company didn’t respond to request for comment.

Meanwhile, the JEDI program faces another legal hurdle as Amazon has also protested the contract and its award to Microsoft last fall. That suit seems a bit more serious, as it’s led to the Department of Defense taking corrective action on the procurement, which will lead to a re-awarding of the contract any day now.

Why CWM solutions offer users a powerful way to streamline work

Chris Aherne has nearly three decades of technology and federal business development experience. He currently leads federal sales for Smartsheet, used by more than 75% of the Fortune 500 to manage and automate their business processes.

It’s hard to understate how much the pandemic has impacted our daily work routines — and the countless rhythms, systems and workflows we all rely on daily to do our jobs efficiently.

Most of us take for granted the underlying IT systems and applications that deliver work to us each day; that help us add value to it and move that work forward. That is, until we can’t get those tools to work correctly.

It’s worth reflecting for a moment on just what remarkable a feat it was that federal IT staffs were able to keep the U.S. government’s roughly 2 million employees and 4 million contractors functioning when so many of them had to suddenly work from new locations, using a massive amount of ad hoc technologies.

Perhaps less obvious, however, was how the pandemic also disrupted the fabric of collaboration that is crucial to how work gets accomplished at scale.

While cloud-based video conferencing and information sharing technologies have helped bridge the communication gaps, they don’t always succeed in providing the collaborative interaction necessary to completing projects and delivering on an agencies’ missions.

That’s due in part because of the traditional ways that enterprises have managed their information, and the types of systems workers have relied upon to do their jobs.

Historically, most organizations rely on large-scale, enterprise resource planning systems to manage their financial information, their customer and personnel profiles, their inventories and other structured data. At the same time, employees rely heavily on office productivity software — spreadsheets, text documents and other applications — to add value to that enterprise information.

It’s estimated that today’s typical knowledge worker concentrates about 40% of their efforts working with structured ERP data and about 60% on more localized, value-adding activity. Where efficiencies often break down is the layer between those two realms — and where the information employees rely on are frequently out of sync.

That’s why a growing number of enterprises are discovering the value of a relatively new category of software called collaboration work management (CWM) solutions.

CWM injects a dynamic workflow layer between an organization’s foundational data systems and the productivity tools employees have become accustomed to using. The real power of CWM solutions lies in the ability they give to employees — and project teams — to design their own solutions and workflows, in a secure, scalable environment.

CWM solutions like Smartsheet and Smartsheet Gov — which was purposely created for government agencies, utilizing Amazon Web Services’ FedRAMP-approved GovCloud  platform — have been gaining a natural groundswell of adoption across federal agencies. That’s in large part because of their ability to:

Provide powerful visibility. CWM platforms can pull together the appropriate enterprise data that employees need, providing a single source of truth; that reduces much of the collaborative friction that typically arises when people use different technology silos.

Enable real-time alignment. A cloud-based CWM platform simplifies the ability for employees to access real-time information at any time and from any location, providing greater collaborative cohesion, faster insights and smarter decision-making.

Reduce repetitive, manual tasks. Many manual processes can be automated, and paper-based workflows eliminated, using a CWM platform. A Smartsheet study found that at organizations with 1,000 or more employees, 59% of respondents surveyed said they could save six or more hours a week if the repetitive aspects of their job were automated; and 72% said the time saved through automation would allow them to perform higher-value work.

Share only what collaborators need to see.  Agencies can still control what information employees and contractors can see and use, thanks to data protection controls that meet federal compliance rules.

Increase accuracy and accountability. CWM platforms help reduce processing errors, increase transparency and improve compliance.

Create configurable reports and real-time dashboards.  CWM tools empower staff to create their own reports and custom dashboards, based on the data they are authorized to access, so they can quickly share insights without having to wait for help from IT specialists.

Improve service levels and savings. The added potential of using automation and artificial intelligence to eliminate routine tasks not only promises to improve citizen services; it also has the potential to help the U.S. federal government free up more than 25% of working hours, according to an analysis by Deloitte.

The use cases for CWM tools are almost limitless. Collaborative work management solutions can streamline acquisitions and contract management; improve budgeting estimates; support supply chain tracking; and help human resources better manage hiring, training and workforce planning activities.

The Department of Energy, NASA and the Defense Department, for instance, are using CWM tools to manage and coordinate large, complex development projects that require constant updates from a multitude of project teams. Smartsheet, for instance, is an ideal collaboration platform for gathering the control documentation and artifacts required for securing FedRAMP certification for a cloud service, as we discovered ourselves.

Perhaps the best reason to try a CWM tool like Smartsheet Gov is it comes with a no-risk trial offer, letting agency employee or workgroup take it for a test drive to see for themselves how effective it can be at streamlining workflows and improving collaboration. There’s also an extensive support system behind it, with our partners at AWS and Carahsoft, and it’s easy to acquire, using GSA Advantage, the online purchasing servicing run the General Services Administration.

If there is an upside to the disruptions caused by the pandemic, it’s having to look for new and better work-arounds to getting our work done. Don’t overlook the power and potential of collaboration work management solutions.

Learn more about how government agencies are evolving to put citizens first.

This article was sponsored by Smartsheet and AWS.

HHS appoints Perryn Ashmore acting CIO, CDO

Perryn Ashmore took over as both the Department of Health and Human Services’ chief information officer and chief data officer in an acting capacity at close of business Friday, according to an agency spokesperson.

Ashmore previously served as principal deputy CIO and had a “close partnership” with his predecessor, José Arrieta, said Eric Hargan, deputy secretary of HHS, in the announcement.

Arrieta, who served as HHS’s CIO since May 2019 and later acting CDO as well, departed Friday, two weeks after announcing his resignation.

“We are immensely grateful for the work that José Arrieta did during his time at HHS, leaving a legacy of transformative advances in our department’s data work, better business practices that will save taxpayers hundreds of millions of dollars, and stronger protections for our department’s networks from cyber attacks,” Hargan said. “In particular, the unprecedented data hub created in HHS Protect will be a lasting, invaluable legacy of Jose’s time at HHS.”

Ashmore is expected to continue that work in a “seamless transition,” Hargan added.

Since late February, Arrieta had been working 18- to 22-hour days including weekends helping stand up HHS Protect, a health platform compiling data from now more than 225 public and private systems to track and limit COVID-19’s spread, according to a person with knowledge of the situation. By resigning he has the opportunity to spend more time with his kids and help them learn while they’re home during the pandemic, the person said.

Arrieta is also spending some of his extra time reflecting on his nearly 17 years in government.

“Over the last three years at HHS, I’ve learned many lessons and failed many times,” Arrieta wrote in a LinkedIn post Monday. “The most important lesson I learned is that a culture committed to innovation and learning from failure is strong and needed to pivot a large entity and combat the largest pandemic in the documented history of humanity.”

Air Force expands its predictive maintenance AI

The Air Force’s fleets of aircraft will be supported by an expanded artificial intelligence platform to predict when planes need maintenance to keep them flight-ready for longer periods of time.

The software comes from C3.ai, a San Francisco-based AI company that has worked with the military since 2017. The company is expanding work it has been doing with Air Force programs, now offering its software suite of AI platforms across the entire Air Force enterprise to reach more aircraft and weapons systems.

The deal, inked with the Air Force’s Rapid Sustainment Office (RSO), includes data science training for the airmen who will use the platform as they deploy predictive analytics on new systems. The contract gives the Air Force the technology and training to be “self-sufficient,” Ed Abbo, C3.ai’s chief technology officer and president, said in an interview.

The platform can handle multiple types of AI and process various forms of input data to determine when a plane needs work before a part breaks. It was first used on older aircraft, which required processing reams of handwritten notes through the company’s natural language processing algorithms. Since then, the company has developed machine learning methods to process telemetry and sensor data from modern planes like the F-35 Joint Strike Fighter Lightning II.

“What C3.ai has developed is a highly flexible platform that can ingest structured and unstructured data,” Abbo told FedScoop.

Air Force’s RSO is not the only office to scale the technology. The Defense Innovation Unit highlighted C3.ai’s potential cost- and time-savings in its annual report in January. And the DOD’s own Joint AI Center has been working on predictive maintenance programs for helicopters. Abbo said the company has also worked with other services like the Navy, Marine Corps and Army’s aviation fleets. If fully implemented across all DOD aircraft, predictive maintenance has the potential to save the Department up to $5 billion annually, according to the company.

“RSO is truly a trailblazer in AI and big data solutions,” Abbo said. “Together, we are successfully modernizing and expanding the Air Force’s AI capabilities that will ultimately extend its competitive edge, support its vision of implementing artificial intelligence at scale, and unlock untold billions of dollars in cost savings by increasing aircraft mission capability.”

OMB details process for removing, excluding threatening tech from federal supply chain

The Office of Management and Budget detailed how its newest council will recommend threatening technologies be removed from federal information systems or excluded from future procurements in an interim final rule released Tuesday.

OMB‘s rule further establishes Federal Acquisition Security Council membership and how supply chain risk information is to be shared in accordance with the Federal Acquisition Supply Chain Security Act of 2018.

Agencies previously worked individually or in small groups to identify supply chain threats and vulnerabilities, but the interagency council will now handle risk information sharing, analysis and assessment.

Based on its assessments, FASC may recommend that the departments of Homeland Security or Defense or the Office of the Director of National Intelligence issue removal or exclusion orders for threatening hardware or software within supply chains.

The recommendation process may be initiated by agencies, non-federal entities or members of FASC, which is chaired by a senior OMB official and comprised of others from the General Services Administration, DHS, ODNI, Department of Justice, DOD, and Department of Commerce.

FASC’s recommendations will be based on 10 criteria concerning the technology it’s evaluating:

• Functionality, including the source’s access to data and information system privileges;
• Security, authenticity and integrity, including that of embedded, integrated or bundled software;
• The source’s ability to deliver the technology as expected;
• Ownership, control or influence by foreign governments or parties with ties to them, especially those deemed adversaries or of special concern;
• Implications to national security and source-critical functions;
• Vulnerability of federal systems, programs and facilities;
• Ability of government to mitigate;
• Credibility of the information;
• Transmission of data to countries outside the U.S.; and
• Other information like the impact to agencies’ missions.

FASC will consult with the National Institute of Standards and Technology to ensure any recommendation it makes is in line with federal standards and guidelines.

Risk information will be shared publicly if a removal or exclusion order is not deemed necessary, but source responses, mitigation proposals and meetings will only be publicized as required by law.

Any of the agencies FASC recommends an order to — among DHS, DOD and ODNI — may issue one, after which GSA and relevant agency officials are responsible for handling all affected contracts.

Agencies may request waivers from orders asking for more time to execute them or exclusion from them in the interest of national security.

FASC will review orders annually and may modify or rescind them.

DHS, acting primarily through the Cybersecurity and Infrastructure Security Agency, will serve as FASC’s risk information sharing agency — responsible for standardizing submission and dissemination. The agency will also manage the Supply Chain Risk Management Task Force, a group of technical experts that will decide how federal and non-federal entities submit information, as well as removal and exclusion order requests, to FASC.

Non-federal entities won’t be required to share supply chain risk information like agencies will, but the removal and exclusion orders will affect their use of products and services. Orders will apply to non-federal information systems when a company is a prime contractor or subcontractor at a federal agency.

FASC will be able to request information from agencies and create program offices, committees and working groups as needed.

The public has until Nov. 2 to comment on FASC’s interim final rule.

How the DOD is developing its AI ethics guidance

It has been six months since the Department of Defense adopted ethical principles for artificial intelligence. Since then, the department’s Joint AI Center has faced the daunting challenge of taking that conceptual work and scaling it to develop actionable guidance for the rest of the military.

The goal is to give anyone who works in technology development — from contracting officers to software developers — a “shared vocabulary” for building ethics into any DOD work involving AI. What’s at stake, leaders say, is ensuring that the DOD uses the emerging technology in ways that uphold the department’s values while managing potentially huge shifts in the “character” of warfare.

The first step is to agree on a document that turns the principles into clear guidance. In the next six months, JAIC anticipates having a detailed guide in the hands of employees across the entire department. The document will explain the DOD’s five ethics principles — that the use of artificial intelligence should be responsible, equitable, traceable, reliable and governable — and describe how they apply in the development of any AI technology, from backend business applications to lethal weapon systems.

For now though, the discussions are still primarily happening within the JAIC, though a small policy team working on ethics-training pilot programs. (The DOD’s AI working group, which is comprised of employees from across the Pentagon, also has a subcommittee focused on ethics.)

Alka Patel, the center’s head of AI ethics policy, told FedScoop her goal is to make “ethics a natural part of the way we all think.” The center is drawing upon experience from several pilot programs, including one that pulled together 15 people who work in different parts of the AI development lifecycle in the JAIC and gave them detailed training to bring an ethical perspective back to their respective teams.

“The JAIC is a wonderful testbed,” Patel said in her first interview with a news outlet in her new role. “We definitely want it to continue forward.” Now, it is just figuring out how to do it best.

Patel said she is in talks with other DOD components and the services to scale the champions program.

Confronting competitors

One of the ways JAIC officials think the DOD can lead in the ethical application of AI is through strong testing and evaluation, said Patel and Jane Pinelis, the center’s head of testing and evaluation. JAIC officials have predicted that the Pentagon will be a global leader in the testing and evaluation of AI due to the seriousness of its initiatives.

The department has to contend with foreign adversaries that may not be as willing to put ethical restraints on their own use of AI in military operations — or at least properly test existing restrains. A global escalation of autonomy in lethal weapons is a top concern for Patel and others in the center, she said.

That pressure pulls the JAIC in different directions — between the need to move fast and the need to be thorough in testing.

“Ethics is not meant to be an obstacle or slow things down or create extra hurdles,” Patel said, but it might seem that way if U.S. competitors and adversaries are putting fewer constraints on their development of AI for warfare.

The JAIC’s testing strategy has multiple angles, including evaluating AI technology against preset requirements, checking its integration with existing systems, examining the human-machine relationship and direct operational testing.

“We have tried to build ethics into every piece of the test and evaluation process,” said Pinelis, who participated in the pilot cohort for the Responsible AI Champion’s program. Few programs in the JAIC have reached the level of maturity to run through all the different types of testing, she added.

Using humans and machines to test

The JAIC will look to both humans and digital tools to build ethics into AI, according to officials. Tools that could quantify how well a model fits a problem set or “remind the human” a model is not suited for certain inputs would help augment users’ judgment, Pinelis said.

JAIC acting Director Nand Mulchandani has stressed he wants to see more testing tools come from the private sector.

“A lot of AI testing is being done manually,” he said last week. There are “not enough tools and products for testing.”

Despite the desire for tools to augment human judgment on when, how and where to ethically use AI, Patel said inside the JAIC, ethical decisions will still rest in the hands of human operators. Even with tools that can show where bias is, or how a model may or may not fit a problem set, the final decision to go or not comes down to the humans in the loop.

“I think about how do we have everyone think about ethics across the organization, and not think perhaps it is just the technologist’s job to address it,” Patel said. “Frankly, it is all part of our jobs.”