Language analysis software aids U.S. Web search for terrorist activity

Computerworld - It's one thing to track and monitor terrorists the U.S. government already knows about. But it's even tougher uncovering the ones who are unknown.
To help in that effort, a Cambridge, Mass.-based globalization software company, Basis Technology Inc., has created the Rosette Arabic Language Analyzer. The tool can plug into data mining applications used by U.S. defense and security agencies that are involved in scouring the Internet for Web sites written in Arabic. By automating the search, information that can help investigators find new potential suspects in the fight against terrorism can be gleaned quickly, according to the company.
Carl Hoffman, CEO of Basis Technology, said the analyzer, which is currently in beta testing, plugs into content management and knowledge management applications used by the U.S. government and defense contractors, including Convera's RetrievalWare and Fast Search & Transfer ASA's Data Search software. It's also available for Microsoft SQL Server and Oracle Text/interMedia. The tool allows the government to focus on obscure Web sites whose URLs change regularly, so the sites can be monitored for terrorist activity.
When interesting or worrisome information is found, it can be turned over to intelligence organizations for direct investigation by personnel who specialize in such probes, Hoffman said.
The analyzer is one of 37 Rosette foreign language analyzers offered by Basic Technology. The analyzers identify the language of the content and then convert the text into standardized Unicode, the international character set that provides a unique number for every character in any language. This is the first commercially available Arabic language analyzer created in the U.S., Hoffman said.
Basic Technology began working on the Arabic Language Analyzer shortly after the Sept. 11 terrorist attacks in the U.S., Hoffman said. "A number of government agencies in the intelligence community strongly encouraged us to move in this direction," he said.
Everette Jordan, director of the National Virtual Translation Center, an organization jointly sponsored by the FBI and CIA under the USA Patriot Act, said in a statement that "linguistics technology is beginning to play an increasingly important role when it comes to ensuring national security."
"Because of the enormous volume of multilingual intelligence information that must be analyzed with limited human resources, technologies that can assist in sifting, sorting and finding critical information are essential in ensuring that threats are detected as quickly as possible," Jordan said.
Glenn Nordin, assistant director of language intelligence policy at the U.S. Department of Defense, said in a statement that analyzers such as this one help because "U.S. government computer systemsare largely designed to work with the Latin alphabet and U.S. character sets, [and] processing information in Arabic is a difficult undertaking.
"In the absence of universal transliteration standards, human transcript of foreign text into the Latin alphabet can result in significant corruption of the data and mismatches in searches," Nordin said. "Finding solutions that enable intelligence analysts to extract and disseminate information in the original language and script could be of critical importance."
Nordin and Jordan could not be reached for additional comment today.

Read more about BI and Analytics in Computerworld's BI and Analytics Topic Center.