Mining for Counterterrorism

As reported by Sarah L. Roberts-Witt,as a result of the events of September 11, 2001, the federal government is paying more attention to the ways that data mining can assist with counterterrorism efforts. At a February meeting in Washington, convened at the request of President Bush's Office of Science and Technology Policy, representatives from IBM, MicroStrategy, and several government agencies discussed their roles in the effort. And in June, the National Research Council presented the findings of a report on counterterrorism technology to Congress, which recommended conducting "more research into machine language algorithms to help the intelligence community mine and combine data." In fact, the council ranks data-mining technology with antibiotics, vaccines, sensors, software, and imaging as technologies that require urgent development in the fight against terrorism.

Usama Fayyad, CEO of digiMine and a participant in several National Research Council panels, explains that data-mining tools can classify interesting occurrences not only in text but also in video streams (for recognizing faces or gaits) and audio signatures in phone calls.

"Identifying clusters of transactions, events, and connected groups that represent travel routes and communications connections is a first step to signal potential threats or activity of interest," says Fayyad. "Often it is not possible to connect such events or entities without predictive components that 'guess' a likely connection—exactly the kind of problem data mining is designed to solve."

This is where text-mining technologies, such as those that digiMine, IBM, Insightful, and SAS are refining, could play a critical role. "Intelligence analysis is about so much more than just numeric databases," says Richard Levitt, senior product manager at Insightful, which is gearing much of its current product development toward the needs of the public sector. "Mining e-mails and scanned documents and then being able to correlate the information are crucial to discovering trends." In the case of homeland defense, it appears that the answers are in much more than the numbers.

Military Intelligence Goes Web 2.0

DOD agency using wikis, AJAX, blogs

The U.S. Department of Defense’s lead intelligence agency is using wikis, blogs, RSS feeds and enterprise “mashups” to help its analysts collaborate better when sifting through data used to support military operations.

As reported by Heather Havenstein the Defense Intelligence Agency is seeing “mushrooming” use of various Web 2.0 technologies, which are becoming increasingly critical to accomplishing missions that require analysts to share intelligence, said Lewis Shepherd, chief of the DIA’s requirements and research group at the Pentagon.

The tools are helping the DIA meet the directives from the 9-11 Commission and other entities for intelligence agencies to “improve and deepen our collaborative work processes,” he said.

The DIA launched its first wiki, dubbed Intellipedia, in 2004 on the Defense Department’s Joint Worldwide Intelligence Communications System, a top-secret network that links all of the government’s intelligence agencies.

“The collaboration potential of the social software side is really being thoroughly vetted and is now rapidly being adopted,” Shepherd said. “Across agencies, wikis and blogs are becoming as ubiquitous as e-mail in terms of information sharing.”

Although the agency’s mission of providing intelligence to support military planning and weapons acquisition is often fodder for spy novels or Hollywood blockbusters, Shepherd said the DIA’s analysts are similar to workers in other industries in that “they rely upon and demand instant gratification” for their information needs.

“One of the virtues of a wiki format is that there is a blurred line between authoring and dissemination,” he added. “The second something is authored, someone else can edit it [and others can] comment upon those edits.”

The agency is also escalating its use of Asynchronous JavaScript and XML, better known as AJAX. The programming method uses JavaScript within the client to build applications that are more interactive than pages built with HTML and that don’t need to refresh a Web page every time a user enters or receives new data.

Last year, the DIA began a project to create a data access layer using a service-oriented architecture to pull together human intelligence (data gathered by people) and publicly available data gathered from the Internet and other sources into a single environment for analysis, Shepherd added.

‘Top-Secret Mashups’

Analysis of data in this new environment will be done in part using Web 2.0 applications, such as “mashups,” which collect data sources such as RSS feeds, Internet maps and information from the DIA network, Shepherd said. Users can access those mashups with a lightweight AJAX front end, he added.

“Web 2.0 mashup fans on the Internet would be very much at home in the burgeoning environment of top-secret mashups, which use in some cases Google Earth and in some cases other geospatial, temporal or other display characteristics and top-secret data,” Shepherd said.

Although he did not provide additional details on how the agency is using mashups, Shepherd did note that the DIA is using JackBe Corp.’s AJAX tools as part of the effort to build this new type of application. JackBe has said publicly that the DIA is using its NQ Suite of AJAX tools to build a desktoplike dashboard that can display intelligence data from a variety of sources through a standard browser.

Prabhat Agarwal, an information security industry analyst at Input, a research firm that specializes in governmental issues, said that the DIA and other defense agencies have become the most advanced users of Web 2.0 tools in the federal government because they have a more secure IT infrastructure.

Department of Defense turns to 'blog analysis'for intelligence

The Air Force Office of Scientific Research, is currently funding a US$450,000 study that attempts to mine blogs for "invaluable help in fighting the war of terror."

How is this going to work? The study's name is cryptic; it's called an "Automated Ontologically-Based Link Analysis of International Web Logs for the Timely Discovery of Relevant and Credible Information" ("Ontologically-based"? Aren't we all?). The three-year project will seek to separate the wheat from the chaff using a radical new approach to information processing: counting the number of hyperlinks that point to a source. As the press release points out, "Within blogs, hyperlinks act like reference citations in research papers thereby allowing someone to discover the most important events bloggers are writing about in just the same way that one can discover the most important papers in a field by finding which ones are the most cited in research papers."

This Brand New Approach™, one with no similarities to that used by the world's largest search engine, will help analysts learn what topics are most popular among bloggers. Basically a Google Trends focused on blogs, the research hopes to clue warfighters into topics that have not yet made it onto the military's radar screen, things like the Danish cartoon controversy that outraged the Muslim world, which was discussed on the blogosphere before it made headline news. Had the US military known about the controversy earlier (perhaps through a hypothetical, full-time US government presence located in every capital city in the world; call it an "embassy," perhaps), Denmark could have been bombed before the situation got so out of control. Or something.

Nate Anderson