FBI seeks software to mine Twitter ‘firehose’

The FBI issued a sole-source justification Tuesday to contract a Twitter alerting tool that would give the bureau access to the “full [Twitter] firehose” — ever public tweet — in real time.

The FBI’s Counterterrorism Division wants to buy at least 200 licenses for Dataminr’s Indicators and Warnings service, which it says is “provides access to directly access the full firehose and it will permit the FBI to design and apply filters that are specifically tailored to operational needs.”

The bureau will use the tool on its unclassified network to “obtain the most current information available in executing its law enforcement and intelligence missions,” its request for quotes says.

“The FBI’s market research indicates that Dataminr is uniquely situated to provide the service the FBI needs – namely a customizable detection and alerting functionality on breaking news and events across all publicly posted Tweets,” the justification says.

Dataminr, according to the solicitation document, “is the only certified Official Twitter Data Partner with direct proprietary access to the full ‘Twitter firehose’ which allows it to provide near real-time alerts of breaking news and events,” a fact the FBI says it confirmed with Twitter.

“Other companies in the market place can provide data analytics of the Twitter feed. However, those companies can only purchase pre-filtered sub-portions of the ‘firehose,'” the justification says. “Unlike Dataminr, they do not have real-time access to the full firehose, nor do they allow customers to customize filters in-house based on changing circumstances.”

Twitter uses a public API that provides access to only about 1 percent of all tweets to the general public and other analytics companies.

“For this reason, data analytics software, which has alerting functionality powered by data derived via the Twitter Public API, is not a viable option,” the FBI says.

Twitter subsidiary GNIP also has this such-sought-after firehose access, but the FBI says the company’s business model doesn’t make sense for its needs of customizable filters and real-time access.

The Transportation Safety Administration and Defense Department also use Dataminr and told the FBI it “was the only service that could satisfy their Indicator & Warning requirements because of the unique relationship between Twitter and Dataminr as well as Dataminr’s ability to provide near real-time access to the full Twitter firehose,” the bureau said.

The FBI will enter a one-year agreement with the company, with two year-long options to extend.