Apple plots touch, voice and gesture UI for 3D iPhone

By Jonny Evans

Apple [AAPL] is working on new 3D-based interfaces for iOS devices. The interface should also be capable of responding to hand gestures, which in conjunction with future advances in Siri will make computing more pervasive, more responsive and always more mobile.

[ABOVE: This video shows it is theoretically possible to combine iPads with 3D technologies.]

The virtual world

The Patently Apple website diligently keeps an eye on Apple patent filings, sharing the cream of these in an intelligent and open-minded manner. It tells us that this month an Apple patent was published which describes a 3D UI for iOS devices. "The new UI will work with proximity sensor arrays and will respond to hovering gestures," the report claims.

Apple has filed a series of these things in recent years, including a 3D GUI for iOS, 3D multifunctional widgets and 3D hovering based gestures for CAD apps on an iPad.

In its essence the idea is that you'll have a 3D display image, but because 3D could be pretty hard to navigate conventionally, the on-board sensors would work to:

"...automatically determine and display a perspective projection of the 3D display environment based on the orientation data without the user physically interacting with (e.g., touching) the display. In some implementations, the display environment could be changed based on gestures made a distance above a touch sensitive display that incorporates proximity sensor arrays." (Source: Patently Apple).

[ABOVE: Who else recalls those reports Apple had aquired military-grade 3D mapping technology firm, C3 Technology.]

We got the tech, how might we use it?

The camera view could be controlled using the device's gyroscope, you'd rotate your device around the X or Y axes to change your view. You'd also have a variety of gesture-based controls to help you navigate within a view.

I find it challenging figuring out how these interfaces will be used,  but here's a few attempts:

-- When used with Mapping and local information, you might be able to use your iPhone for access to interesting augmented reality applications.

-- When used in games, you can look forward to interesting new experiences in 3D. If used on the Apple TV you can expect the Apple effect to impact 3D TV, a much-championed feature which hasn't really had its day, possibly because the 3D movies released so far are utterly, utterly boring. Oh, and the iPad 3 will very likely be HD, and potentially 3D.

-- If extended to the Mac, you can imagine a 3D desktop experience, in which active apps and open windows nestle behind or beside each other -- to bring them to the front, all you may need to do is point.

-- Returning to the Apple TV, touch and voice could become the remote control, with Wii-like gaming and other experiences to be had from the Ive-designed goggle-box.

Interesting times

It's interesting. When I think about some of the things we take for granted today when using technology in comparison to what we once expected, it's pretty magical. You don't have to be a rubber-clad dancer to feel the reality shift taking place.

Should we expect Apple to deploy a 3D interface soon? I don't think so. I anticipate the firm will deploy such technologies in its usual incremental way. So you can predict a slow introduction of these new UI elements, beginning potentially with the next major release. Of course, merely because Apple has a patent on this tech doesn't mean it will deploy it, though it will if it sees a good reason to.

In the case of iOS devices, I'll stretch my arm and suggest we may see a 3D interface introduced within Apple's future Mapping application. This will then be gently extended across the device UI, but only slowly and only where it makes logical sense. Apple will then build out from there.

Of course, these 3D user interfaces will require a huge amount of processor and GPU power. I wonder how Apple intends resolving that tricky problem?

Got a story? Drop me a line via Twitter or in comments below and let me know. I'd like it if you chose to follow me on Twitter so I can let you know when these items are published here first on Computerworld.       

Computerworld's IT Salary Survey 2017 results
Shop Tech Products at Amazon