Is it reality or is it augmented?

Augmented reality is becoming a big deal. What started as an interesting, albeit geeky, application of technology is starting to find real-world uses that are impressive, not to mention way cool.

AR is the technique of inserting virtual objects into real-time video to create the illusion that the virtual objects are part of the scene. To do this requires analyzing the video to determine the geometry of the scene, warping and modifying the virtual object (or objects) to conform to the scene's perspective and other attributes, and then rendering the virtual object(s) in their required positions on the display.

There are three main approaches to analyzing a scene to figure out where to place the augmentation: The first technique is to rely on metadata.

For example, if the video is being provided by a smartphone such as an HTC G1 or an Apple iPhone, the GPS, compass and inertial sensors can be used to determine the phone's location and orientation. This is enough data to determine where the device is geographically located, where the horizon is in the scene, and which direction the device is facing.

An example of this approach can be found in the application Wikitude. This program (now available on both the G1 and the iPhone) overlays the video with markers to point out landmarks and points of interest that are cataloged in a crowd-sourced wiki.

A similar approach is used by Google's SkyMap for the G1, which actually replaces the entire scene with an overlay of the star field that should be in view, giving you an instant mobile planetarium.

The next scene analysis approach is to use a physical "target" in the real world that the AR software looks for, the most common targets being simple patterns printed on paper. Once detected, the target's orientation and geometry can be determined and used to position the virtual objects in the scene.I discussed an impressive example of this technique in my Network World Web Applications Alert newsletter last week. The U.S. Postal Service (USPS) has an online AR application, the Virtual Box Simulator that is really clever and actually does something sort of useful: It shows you whether a real-world object can fit in one of their standard shipping boxes.

You print a target image from a PDF file the USPS supplies (the target pattern for this service is, not surprisingly, its eagle logo) then load the Virtual Box Simulator Flash-based AR application in your browser. This application requests access to your camera and when you show the camera the target, the application overlays it with an image of one of the boxes. You can select the transparency and which size of box and by moving whatever you want to ship onto the target you can determine whether the box is large enough.

I have one big complaint about the USPS AR application; in fact, it's the same complaint I had with General Electric's "Smart Grid" AR application they published a while ago: They refer to these as "holograms," something that they absolutely are not! If you are uncertain of the distinction, see Wikipedia for a pretty good explanation.

Anyway, when I wrote above that the USPS Virtual Box is "sort of useful" it's because the AR application doesn't quite work: It doesn't know enough about the geometry of the thing you're trying to fit into the virtual box. So, if that thing extends, say, above the surface of the box and points towards the camera, the part of the box surface that should be obscured will actually overlay the thing.

The final technique for determining scene geometry is to analyze real-world features and attempt to determine either the overall scene geometry without needing an AR target or to work "around" objects in the scene (the latter is what the USPS AR application would need to do to correctly deal with real-world objects that extend beyond the virtual box boundaries).

This is a technique that I have yet to see used in a commercial application and it requires much more sophisticated analysis algorithms which, in turn, requires a lot more processor power than you'll find in most mobile devices and, indeed, many PCs.

In fact, processor power is one of the bigger problems of AR applications because, while real-time scene detection is tough enough, the real-time rendering of one or more virtual objects that have more than simple geometries and use animation, is a much harder. In fact, you can see how hard AR is in the USPS application: The frame display rate often drops noticeably when you move the target.

I'm very impressed with the potential of augmented reality and despite the loose terminology used by the USPS and GE use, these early applications are really impressive. So, do you think your company be using AR?

This story, "Is it reality or is it augmented?" was originally published by Network World.

Copyright © 2009 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon