Opinion: Transparency is clearly on the way

Computerworld - The perfect storm of transparency is gathering over today's enterprise computing environment. Today we are seeing the gradual demise of stovepipe applications, a rapid decline in the cost of enterprisewide (and Webwide) queries and a stunning proliferation of data from sensor networks inside and outside the enterprise. Add to this the fact that storage costs seem to be approaching zero, and you have the makings of a transparent enterprise.
In a transparent enterprise, we will be able to know more about the disparate data sets that we create, find unexpected connections in previously unconnected pieces of data and see more information about individuals, whether they are customers or employees. Transparency will emerge through business intelligence systems, in buildings whose security systems record and log the identity of anyone in the facility at any given moment, and in collections of telemetry data from various private and public wireless networks that tell us the location of individual devices and their users.
Consider the enterprise data store that not only has detailed sales transactions but also travel expense reports such as rental car records, cell phone billing details, meal receipts and perhaps even electronic toll-booth records. Of course sales managers will be able to determine which sales rep has the most efficient sales-to-expense ratio. But managers may also be able to figure out that an employee exceeded the speed limit because he wrote a sales invoice in Albany, N.Y., just two hours and 10 minutes after leaving Burlington, Vt. Rental car records and MapQuest will confirm that it should take two hours and 18 minutes to drive that distance at the legal speed limit.
If you've got a database integration or business intelligence project that's behind schedule, you're probably skeptical about whether this magic that I've dubbed transparency will ever arrive. Looking past near-term sticking points, we can, however, be confident that it will. We're collecting more and more information, and it's easier to just keep it all than it is to figure out what to delete. Structured data formats such as XML and RDF are making it possible to mine this growing volume of data for new information. Perhaps most important, the systems we build today will one day support queries that we didn't even anticipate.

Changes in the public policy environment are also leading us toward greater transparency. Regulatory requirements such as the Sarbanes-Oxley Act and a California law requiring disclosure of security incidents demand greater systems transparency. Privacy advocates such as science fiction writer David Brin are even calling for increased transparency of surveillance