Memex: Law enforcement’s answer to searching the dark Web
Those committing crimes under the anonymity of the dark Web are beginning to see the ease of their transgressions erode thanks to the Defense Advanced Projects Research Agency.
DARPA displayed the Memex search engine during its “Wait What?” Conference in St. Louis last week, showing how the technology pushes the boundaries of what can be indexed in the dark Web.
Launched earlier this year, Memex brings together 17 different contract teams that have been working with DARPA to create new forms of discovery, organization and presentation of domain-specific content. Memex is searching for content hosted on sites with a .onion domain (which can only be accessed with a TOR browser) or other Web pages that are ignored by commercial search engines. The tool goes beyond what is catalogued by commercial search engines like Google or Yahoo, which only index around 5 percent of the Internet.
Wade Shen, DARPA’s program manager for Memex, said the engine was built to get the answers law enforcement agencies need when searching the dark Web for illegal activities.
The dark Web “is generally not where you get interesting questions answered,” Shen said. “If you want to know what the population of China is, you don’t go to the dark Web for that thing. If you want to buy, say, illegal drugs — that is where you want to go.”
While Memex does not remove the anonymity that comes with the dark Web, it gives law enforcement a fine-tooth comb to search for illegal activity. A demo shown to FedScoop at last week’s conference was granular enough to show the message board conversations around one person in the European Union who was claiming to sell illegal firearms. Also, DARPA showed how it could tweak a graph to measure traffic related to a subject, ebbing and flowing over the course of a few years.
Memex’s engine allows users to cross reference traffic from individual users based on a number of different inputs for a granular look at how traffic has progressed. (Greg Otto/FedScoop)
“We can understand the distribution of things, the composition of these marketplaces, decompose them in the number of users, when there is upticks, when there is downticks, what happens when law enforcement shuts down a marketplace,” Shen said.
The search engine has already aided human trafficking investigations in New York City.
John Temple, chief of the Human Trafficking Response Unit for New York County, said Memex helped the New York Police Department investigate the online footprints human traffickers were leaving that were ignored by commercial search engines. Using different filters in Memex, investigators could index different ways traffickers avoid traditional search engine indexing, like posting photos of phone numbers or spelling out the entire 10 digits.
“We were going into a commercial search engine and then see 10 blue links,” Temple said. “That wasn’t getting at the information that was being obfuscated by the traffickers. [Memex] takes all of this information and looks at it in a way that we historically haven’t been able to do in an effective way.”
Shen admitted that Memex won’t exactly pinpoint who is responsible for proliferating such crimes, but the ability to index what’s going on is a far cry from leaving police to operate by traditional means.
“Often times, adversaries don’t care that we are doing this,” Shen said. “They think they are anonymous, and they are. But we do have a strong motivation to find out what’s going on under the hood. Maybe a target is someone we should be tracking online so that we can predictably decide the next illegal transaction and the future investigative processes of crimes.”
Reach the reporter at greg.otto@fedscoop.com or follow him on Twitter @gregotto.
