Mixed Reality Changes How Engineers Work

A bridge spanning the Delaware River that connects the New Jersey Turnpike with the Pennsylvania Turnpike was abruptly shut down earlier this year when bridge inspectors on site discovered severe structural deficiencies. New mixed reality tools and equipment could change all of that. Rather than working on site, engineers or inspectors will be able to view the bridge remotely through a visual headpiece to identify problems or collaborate with workers on site in real time. Using mixed reality tools, engineers in other disciplines will also work differently as the tools upend the design process.

Holographic computing is the gamechanger, and Microsoft’s Hololens is the vanguard of the new order. The device is a wearable holographic computer operating on a specialized version of Windows 10. It enables users to interact with 3D holograms generated from a separate processing unit that calculates where 3D graphics exist in the user’s physical space. Users can pin holograms to physical objects using gesture, gaze and voice commands.

“This bridges the gap between digital and physical,” says Aviad Almagor, director of mixed reality with Trimble, a Sunnyvale, CA, developer of positioning products. The company was one of the first to develop an app for Hololens, called SketchUp Viewer, a version of its existing 3D modeling platform for architects and engineers. “Its value in the field may be bigger than in the office. It completely changes the way you work on site.”

As 3D modeling has matured, engineers and researchers have been looking for methods to move 3D models from the tether of 2D screens and desktop or even laptop computers. The development of virtual and augmented reality met that challenge—to a point. Using headsets, virtual reality programs immerse the user in a virtual setting where you are not aware of the real, physical world around you. Augmented reality enhances the world by overlaying digital information on top of vision.

Civil engineers, manufacturing engineers and others have been using virtual reality to design things like buildings and cars on monitors or large screens for some time. Firms have invested in CAVE technology, where design teams can view virtual images projected on the walls of a room and make design changes virtually, saving money on physical models and speeding up design.

Hololens improves on that, allowing the user mobility and the ability to collaborate remotely. Unlike virtual reality, it projects images onto the lenses where they are perceived to exist together with real-world elements. The images are interactive; using voice or hand commands, users can alter the images, or extract layers from within an object.

“As a technical achievement, this is by far the best,” says Ioannis Brilakis, director of the Construction IT Laboratory at the UK’s University of Cambridge. Brilakis and his team worked with Trimble to develop two Hololens apps for the engineering/construction industry. While engineers have embraced technology such as building information modeling (BIM) and more, he says contractors and others have been slow to adopt new technology.

That is changing, and rapidly, he says. The apps his team has developed target bridge inspectors and engineers. The first, Project Monitoring, projects 4D design models and places them over the actual structure as it is being built. It allow engineers and contractors to visualize progress and find errors during construction, such as building elements that are missing or improper placement of pipes, wires or doorways. An onsite inspector can view the models through Hololens and identify issues and solutions while an office-bound engineer fitted with the device sees the same image and advises.

The second app is for bridge inspections. Workers at the site take photos of the entire structure using digital single lens reflex cameras or laser scanners and upload the images to a cloud service. From there, engineers import them to the Hololens, which stiches the images together to build a comprehensive model of the entire structure. Using the Hololens, engineers or inspectors in the office can call on the full image, extract layers or details, and walk around the hologram to detect structural anomalies.

Using the app will reduce costs and time needed for accurate inspections, says Brilakis, pointing out that the ability to zoom in on different parts of the structure as well as small details not easily accessible for physical inspection promise better diagnosis. “One question is, “How much accuracy do you need?” he says. “We’ve got it down to plus or minus five millimeters accuracy. We could get it to one millimeter. Do you really need that?”

“This is a new wave of technology,” says Almagor. “From PCs to the internet to mobile devices, we’ve consumed data in the same way, on a 2D screen. 3D is now a must-have tool, from mining to manufacturing. For the first time now with mixed reality, you can take the content out of 2D and into the physical world.”

Other industries are beginning to use mixed reality for training, education and design. Microsoft approached a handful of firms and universities during development to explore how Hololens could be used. Japan Airways, for instance, is using it to train maintenance personnel on servicing jet engines, and ThyssenKrupp is using Hololens to troubleshoot and repair elevators. Stryker, a medical technology firm, is using it for operating room design.

Mixed reality also is making its way into education. Pearson recently partnered with Microsoft to develop learning tools such as a pilot program at Texas Tech University and San Diego State University for nursing. Schools often used actors to mimic real-world medical situations as part of training. The program would help standardize or replicate these situations using holographic video tools and actors.

“There’s no such thing as hardware without limitation,” notes Brilakis. He thinks the headset is a bit heavy and front-weighted, and battery life can be improved. “If you use it for eight hours of work, you may need something different. But this is just Version 1. There will be more to Versions 2 and 3, and is should be a lot more robust than the current iteration.”

For Further Discussion