The U.S. Department of Homeland Security has a vast collection of data that can help local, state and federal agencies plan for and respond to emergencies. But the disparate nature of this data, which includes more than 100,000 facility records, 20,000 documents, 50,000 images and 5,000 Web pages gathered from various government entities, made searching for specific information and insights impossible.
In October 2013, the Idaho National Laboratory took on the challenging task of using all that data to create its Infrastructure Protection Digital Library.
"Our job was to normalize the content so it could become searchable and useful," says Wayne Simpson, an architect in information management at the lab.
INL photographe r Chr i s Morgan Idaho National Laboratory’s big data team: Wayne Simpson, Mary Klett and Dale Christiansen.
Simpson says the project required him and his colleagues to find ways to deal with the variety of ways people store the information -- in databases, as documents, on Web pages or as PDF files. Working with data feeds and crawlers, team members built a system that could access data however it was stored. Then they normalized the data and pushed it into search tools. They're now adding tools that can help users analyze data and predict how emergencies could impact infrastructure in their regions.
Michael Norman, director of the DHS's Infrastructure Information Collection Division, says the library saves people hours of work by making information easy to find. It also helps ensure that government officials don't miss a critical piece of data.