Compressed data search tech goes mainstream

Computerworld - After selling its data compression and search technology to the auto industry for six years, WindSpring is eyeing the storage market with a compression management technology that can find information while the data is still compressed.

The company was just awarded a U.S. patent (download PDF) for its Data Management Tools (DMT) product, which allows direct access to compressed data sets without the need for decompression.

WindSpring CEO Tom Hunt said by the end of this month his company will begin shipping products to storage equipment manufacturers and other firms. Hunt sees WindSpring's offering as being particularly valuable to the mobile market, where the more data you can store in a smaller space the better. That typically means compression is needed.

Hunt claims most popular compression algorithms, such as PKZIP, haven't changed much since the 1980s. They provided good methods for reduced data storage requirements, but offered no efficient way to search the data -- other than to first restore it to a primary system and then decompress it.

"It's literally the same codec that's been around 20 to 30 years," he said.

At a high-level, WindSpring's flagship DMT software development kit (SDK) compresses data and indexes it at the same time; when a data search takes place, only the exact data requested is retrieved and decompressed.

"Our encoding mechanism creates a table where we know where everything is always. So when a search engine says 'find this information,' we go right to it," he said. "So it's instantaneous access, with very little overhead."

If a company is already using backup or archival software that relies on an off-the-shelf compression SDK such as LZMA or LZJB for backup and database compression, WindSpring's universal decoding capability can take ir or any industry standard compression algorithm, read it into its system and encode it for later searches.

WindSpring has been delivering its compression technology to the embedded market, including navigation systems in for several automakers including Mercedes Benz, Toyota and Lexus.

Along with the mobile market, WindSpring has its sights on a wide range of potential original equipment manufacturers, such as IBM, Oracle, EMC, HP, and NetApp, as well as disk drive makers. They could include the software in their products, helping corporations with regulatory compliance and legal discovery requests, Hunt said.

"You may have a DB2 or Oracle application and need to optimize your storage, but you still need to be able to search it, and you can't keep building bigger and bigger data centers," he said.

Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian or subscribe to Lucas's RSS feed . His e-mail address is lmearian@computerworld.com.

Read more about Storage in Computerworld's Storage Topic Center.