Forget augmented reality (AR) and virtual reality (VR); mixed reality (MR) is set to change the way we think about how the real and the virtual interact.
MR is the merging of the real and virtual worlds to “produce new environments and visualizations where physical and digital objects co-exist and interact in real time” and has applications across a range of areas from communications to entertainment and logistics. Proponents argue it can extend the Internet of Things (IoT) in new directions, too.
Despite recent hype, there continues to be a debate about what MR really is, whether it exists at all and what companies like Microsoft are doing with it.
What is mixed reality?
Daniel Eckert, PriceWaterhouseCooper’s Managing Director of Emerging Technologies, believes that, unlike AR, MR requires a Head Mounted Display (HMD). Rob Minson, CTO at Fracture Reality, explains that MR combines the immersive capabilities of VR – a head-mounted optical device rather than a small mobile phone window – with AR’s ability to bring data into the physical world. This powerful combination makes it a medium suited to both consumer and enterprise domains and marks a new paradigm in the way people interact with today's data-rich world.
For Microsoft’s Greg Sullivan, Director of Communications for Windows and Devices, mixed reality describes the continuum between the physical and digital worlds, from augmented reality to virtual reality and everything in between.
“Mixed reality blends these worlds to produce new environments where physical and digital objects co-exist and interact in real time. Bringing together the digital and 3D physical worlds enables us to do impossible things and is the future of computing,” he says.
Gil Shefler, marketing manager and VR evangelist at Kaltura, says that the line between augmented and mixed reality is blurry and at one pooint they referred to the same thing. Increasingly, AR refers to an “annotated” reality, Shefler says, drawing on the ideas of futurist Jaron Lanier who defined it as: a medium where things like texts and graphs are projected on a person’s field of vision providing information on their surroundings.
“MR is when whole images or objects are projected, mixing virtual and real worlds and the two interact,” Shefler says.
The lack of a killer app or product – the expected release of Magic Leap’s device this year may change this – means that there is no clear leader as yet, although a number of developers have staked their claim, most notably Microsoft.
What could MR be used for?
There are already a number of applications of MR in numerous business sectors. Ford, for example, is using HoloLens to unleash creativity by allowing designers to experiment with prototypes without having to physically build every design in clay. Similarly, businesses like thyssenkrupp are using it to drive sales by enabling reps to provide customers with a visualization of the stair lift they’re looking to install in their homes.
Sullivan adds: “By transforming their business processes, thyssenkrup are able to streamline internal processes whilst also helping their customers make a significant addition to their home with much more confidence.”
The ability to blend digital information and objects seamlessly into the physical world could change the way people use computers. Perhaps most disruptively, Minson explains, MR will make physical screens a thing of the past.
“Want a 60-inch plasma screen on your wall? Just put it there. Now want that same display pocket sized and mobile? No problem,” he says.
He also believes that MR will change how we collaborate. The natural way for humans to interact is to gather together in a shared space, to talk and to gesture. Traditional 2D screens and video conferencing are a massive step backwards in terms of communication and understanding, but when it comes to digital information they have been our only option until now.
“MR allows us to collaborate both locally and remotely around digital objects and data using the same human tools we use for physical objects. These could be interactive architectural models, medical scan data, or 3D data visualizations,” he says.
For retailers, Shefler says, mixed reality means a display at a store might always be full. Computer-generated images of missing items can be projected in their physical absence. In fact, new items that have not yet been released can be on display using graphics.
“Taking it a step further, virtual salespeople might replace real ones – a combination of AI and MR,” Shefler says. “You might be able to look at furniture at a store and project it onto your room to see if it fits.” Ikea is already doing similar things.
Eckert says much of the application of MR lies in the Field Service, manufacturing and training areas. He believes empowering field service personnel with real-time data (and objects) is the killer app for AR/MR.
He also sees great potential in 3D analytics and visualization. Data visualization has been a challenge as it is limited to a bi-planer experience. “Putting the data in 3D and flying and/or manipulating it in 3D enables data scientists (and the Joe Sixpack for that matter) to look at data models in completely new ways.”
Minson adds, by scanning the world around us, MR devices can superimpose data in position over the top of objects or physical spaces. This capability has huge implications for empowering workers in complex, data-rich environments such as engineering, complex manufacturing or emergency response.
Zappar, for example, recently worked with Kindred Group to explore data visualization in high-profile sporting matches using the 2017 Champions League final and a T20 cricket match between South Africa and England as examples. Users were able to explore the team’s players, with insights into their goals, shots or runs – interacting and unearthing new stats while putting the user at the heart of the data.
Sullivan believes that companies that embrace mixed reality as part of a broader digital transformation will unlock its full potential. “Understanding that mixed reality can complement a business’ transition to cloud or its integration of AI will empower them to use mixed reality to foster remote collaboration and innovative creativity.”
For Ajay Singh, director at Samsung NEXT Ventures, one of the investment arms of Samsung, the key to leveraging the potential of MR is to ask what real problem you are trying to solve, and whether MR can really provide a solution. He cautions that mixed reality can’t be approached as a flashy marketing ploy; it must align with the core values of the company and provide real value.
“We haven’t seen a remarkable use case for mixed reality on the market yet, and we are still in the early phases of the technology’s life. But it’s not difficult to see the possibilities,” he says. “The largest hurdle is to define the correct use case for the technology that impacts a company’s core values. There are examples of early adopters of mixed reality out there now that provide very basic level understanding of the potential of the technology and have seen user engagement grow as a result, and I expect engagement to become more and more of a factor as the technology becomes more accessible.”
What do businesses need to understand about MR?
Eckert emphasizes the need for those businesses wanting to invest in MR to really understand what mixed reality is and isn’t and to cut through market confusion. “It’s critical that decision makers understand the difference between AR, VR, and MR – and most importantly when and where to use the right technology to solve the problem they are addressing,” he adds.
In addition, a multi-disciplinary team is vital to the success of an MR project. Eckert says that while you could argue a design agency, game studio, creative team or a software development team, for instance, is all that's needed to create an MR solution, that might be okay only for a simple proof of concept. If you are thinking of deploying MR in the enterprise, you need to add many more skills to the equation, he says, including:
- Security: If a company is going to deploy 1,000 HMDs at a cost of over $2.5 million into the field, there needs to be security, provisioning, asset tracking and central management.
- Infrastructure: Most of the HMDs are tethered – and network connectivity on a factory floor is challenging – even in the best of factories. Companies need to consider network connectivity, latency, data encryption, storage and battery life.
- Ergonomics: Wearing an HMD for more than one to two hours can be exhausting and disorientating if the HMD is too heavy, hot or distracting. The lenses might get dirty or the HMD might not work in low light. “It took 34 years for industrial designers to get the feel of the mobile phone right – my point for MR HMDs – it won’t be accomplished overnight,” Eckert says.
- User experience: Companies need to consider how the interaction with MR content will take place, whether by gestures, voice commands, joysticks, gloves, or wands, and then they need to train users how to interact with their environment as there is no standard UI currently. Bringing voice into the equation adds a whole host of new problems from noise, wind, external voices, language barriers to context and local vernacular. All this must be designed and managed to get the most out of the system.
Addressing these aspects in addition to the core content design will aid the success of an MR project. Content, Eckert says, is really king. And creating that content requires a strong understanding of art, programming, 3D design, modeling, simulation, AI, lighting, context/intent and physics. “If you do one thing – learn as much as you can about how to create, build, and manage content for the MR world – because no matter the hardware you select, the objects you create can be repurposed to run on different platforms,” he says.
Eckert adds that just because you can, doesn’t mean you should invest in MR, adding that for now – less is more.
David de la Pena, head of Technology at digital agency DARE West, also cautions that the tech is still maturing. “It’s easy to get caught up in the hype and novelty of any new technology; ...be prepared to face a number of barriers and problems to solve. ...We’re in a period of exciting experimentation. So for any company looking to invest in MR, they need to understand that they won’t see results right off the bat.”
De la Pena adds that the priority should always be added value to the end user, rather than simply a great demonstration of the technology at work. “Real use will come only where it makes sense - where it brings true value. For the best outcome, up-front user research is an essential part of the process for this,” he says.
This story, "What you should know about mixed reality" was originally published by IDG Connect.