When there’s a need for an expert to inspect an oil rig or train someone to repair a vehicle, companies used to fly the expert in to do the task in person. More and more, though, companies are learning that when workers in the field are equipped with assisted reality wearables, the expert can help them from a remote location.

The “see what I see” capability of these devices with a head-mounted camera and display allow the expert in a remote location to see what someone else is seeing on site. The expert can also give the person wearing the device verbal instructions and visual instructions through the display. In addition, the person in the field can use the device hands-free.

This kind of “remote expert guidance” has been a common use case for RealWear equipment since the company’s founding five years ago, but it was accelerated when the pandemic prevented people from traveling. Now that more companies have experienced the alternative, and the savings in travel costs and experts’ time, many are likely to stick with it.

“We think this is the new normal, that not everybody needs to travel all the time,” Rama Oruganti, chief product officer at RealWear, told PYMNTS. “There are certain tasks that can be done remotely as long as you have the right point of view, you can see the things and you have the tools in place to make remote work possible in that way.”

Providing Information Without Distracting From Hands-On Work 

RealWear launched the latest generation of its industrial-strength wearables on Dec. 8. Assisted reality wearables like the new RealWear Navigator 500 incorporate the digital world but do not immerse the user in it or put it in the user’s field of vision. Instead, with assisted reality, the digital world is right below the user’s field of view, so they need only look down to see it — just as the driver of a car would glance down at the dashboard, Oruganti explained. In industrial uses cases that often include hazardous environments, it’s important that the user’s field of view not be obstructed, he added.

“Industrial frontline workers are the people who can use most of the helpful things that might come out of the metaverse,” Oruganti said. “So, we are trying to take those and put it into the hands of the people who need it the most.”

The RealWear Navigator 500 is two-thirds the weight its predecessor, so it’s easier to wear during an eight-hour shift; it’s two-thirds the width of the earlier device, so that the center of gravity is closer to the user’s temple and the perceived weight is lower; it’s rugged enough to be dropped from a height of 2 meters without damage, and it’s modular so the camera and display can be changed in the future.

Enabling Digital Workflow and Visual Assist 

Another growing use case for these devices involves inspections in which the user is on site and doesn’t need remote assistance. This “digital workflow” use case applies the product’s ability to respond to voice command and to record what the user sees. While inspecting equipment, for example, the user can mark off items on a checklist with verbal commands and can record what they’re seeing. Previously, this would be done by marking the items off on paper and later entering that information into a laptop.

“That was one of the big things we resolve,” Oruganti said. “One, you reduce paperwork errors; two, you save time, and three, you have visual documentation.”

A third growing use case for this equipment is “visual assist.” If someone is repairing a piece of equipment and needs to see the blueprint as they work, they can see it on the display on the wearables. What’s more, they can track around that piece of paper and zoom in on the part they need.

“It’s like someone’s holding a seven-inch tablet at arm’s length with that information for you, so that’s very helpful,” Oruganti said.

Displaying IoT Data When It’s Needed 

Assisted reality wearables can also display information from the sensors on a piece of equipment so that the user can see if a machine is too hot or spinning too fast, for example, without having to read the dial. The Internet of Things (IoT) data is funneled into the display and the user can see it when they want it.

“[Assisted reality] and [virtual reality] are going to be big — they’re already big and they’re growing fast,” Oruganti said. “The big immediate use case for a lot of people in the real world is going to be in industry. There are 100 million industrial frontline workers — the kind of people we are targeting — so we are trying to take the best in class of things that are applicable here without being distracting and without taking away their hands.”

In 2019, the European Manufacturing sector was responsible for an annual total of 880 million tons of carbon dioxide. This makes it one of the largest single emitters of greenhouse gases in Europe. While in the US, manufacturing accounts for almost a quarter (23%) of direct carbon emissions

With the current global spotlight being shone on climate change, countries and large corporations are having to agree to new climate regulations in the bid to reduce their carbon footprint, and are looking to innovative and complementary technologies that will help to reduce emissions, whilst future proofing their operations – including the adoption and deployment of extended reality (XR) technologies.

What is XR

XR (or eXtended Reality) is an umbrella term that encompasses the immersive Augmented (AR), Mixed (MR) and Virtual Reality (VR) technologies.

Understand the differences between AR, MR and VR

XR technologies allow data (for the point of this post, 3D CAD and PLM data) to be interacted with as a 3D digital representation in context, and at full scale.

How can XR make it green

On the surface, using XR devices may not seem like they are making much of a difference to an organisation’s green credentials. But small changes in the early stages of product development can filter through along the product’s lifecycle, from the initial design stages to the finished product rolling off the production line.

So where to use it…

Design reviews and prototyping

The design review process often starts with teams looking at a CAD design on a computer screen. However, you never quite get the full picture through a 2D monitor- it can be hard to imagine the true shape and scale of a product, and whether the ergonomics of a design will translate to a physical product.

Usually the next step is to produce 3D models of the product, sometimes at full scale or sometimes as a smaller representation. But either way, additional materials such as clay or foam are required. If the modelling phase needs more than one iteration of a model, then the amount of material used (and the waste product) adds up. And what happens to the models when they are finished with? Do they just get thrown away or are the materials repurposed?

By using XR technology, design reviews can be conducted in context and at full scale, giving designers and engineers the chance to make changes without having to build additional models. The changes can be made in the CAD system and then re-checked in the XR device.

This saves on physical design iterations and therefore the need for modelling materials, which in return reduces the waste produced as excess.

Training

XR is great for training. By practicing a process (such as the assembly of a component) virtually, if errors are made, materials are not wasted and machinery is not needed to be used. The training process can be repeated as many times as necessary. So when the operator is experienced enough, the materials and machinery should only need to be used once.

Collaboration

Today’s products are rarely developed with the luxury of co-located teams.  Globally distributed design is practically unavoidable, and increased home working is further adding to the challenge on effective collaboration between design teams.

By using the collaboration capabilities of XR to connect with globally located teams for processes such as design reviews and factory layouts, the need for previously required travel (including flights to visit various facilities) is redundant. With air travel being a major contributing factor to CO2 emissions, taking that need for travel away will make a difference to your green credentials (and your wallet!).

Starting small is still a start

Obviously there are changes that need to be made to manufacturing process on a grander scale in the attempt to cut emissions and waste, but these projects could take years to implement and cost a lot of money. Switching to XR for some smaller elements, like those mentioned above, may not make a massive dent in the fight against climate change, but its a good place to start.

Perhaps the most promising XR technology applications are found in manufacturing and industrial environments. Indeed according to PwC research, the use of VR MR and AR in product development could raise GDP by $360 billion by 2030.

When determining the optimal deployment strategy for these technologies, manufacturing and industrial business leaders should consider the following:

1. XR has the potential to improve the product design process
2. XR tech can aid in the process of production planning
3. AR is suitable for assembly lines

Examples are given in the article about immersive XR use including Google Glass and Microsoft HoloLens with examples from workers at GE. The original article can be read here.

Human ability to imagine objects that are not physically present is limited. It is even more difficult for us to mentally place them in an existing environment. How often, for example, has it happened to you that a newly purchased piece of furniture was too large for the intended space?

In construction planning and architecture, this problem is amplified. Whereas in the case of the previously mentioned piece of furniture, only a single part has to be inserted into an existing space, in architecture we are often dealing with entire buildings in which floors, rooms and objects stand in a relationship to one another; and of course, the building itself as a whole must also fit into its surroundings. In this process, our lack of imagination can lead to mistakes with far-reaching consequences.

This is where technology helps our imagination tremendously. Augmented reality (AR) in combination with Building Information Modeling (BIM) ensures that we can “actually” see all objects and relationships.

What is BIM? What is AR?

Building Information Modeling (BIM) is to be understood as a digital method that is used throughout the life cycle of a building. In this process, all data and information related to the construction is stored and mapped in a BIM-enabled software.

Augmented reality is the computer-aided expansion of reality perception. Specifically in construction planning, the BIM models are “projected” into the real environment.

Benefits of AR in Construction Planning

AR applications for construction planning help our imagination tremendously. They support the entire decision-making process both on the side of the construction planner and on the side of the client. AR glasses can be used to better present and understand the planned building. Thus, decisions in the early planning phases can be made more easily and more correctly, which reduces planning and construction costs accordingly.

“Especially in the case of existing building conversion, it is advantageous if you can visualize the superimposition between the model and reality.”

DI Dr. Timur Uzunoglu, Managing Director convex ZT GmbH

AR Use Cases in Construction Planning at convex

At convex ZT GmbH, we use AR technology from the design phase to operation. With Holo-Light’s AR3S software, we bring BIM planning closer to clients and enable greater planning transparency. Building owners feel more involved in the planning process during our AR-assisted planning meetings and can make better decisions. We make AR inspections together with the builders directly on site. These AR inspections provide a direct impression on site in real time and help to weigh alternatives against each other. In revitalizations of existing buildings, it is often challenging to bring the new structures into a functioning harmony with the existing buildings, and AR helps very well there, too.

The Ministry of Defence said, “The operation was performed by a medical team headed by Colonel Doctor Ahmed Al Jahuri, senior consultant and head of the Orthopaedic Department at the Armed Forces Hospital, and medical staff from the orthopaedic, anaesthesia, operations, radiology and medical engineering departments.”

Speaking about the surgery that employed cutting-edge technology, Dr Al Jahuri said, “The Armed Forces Hospital was able to use state-of-the-art technology in augmented reality as a solution for the development of minimally invasive spine surgeries.”

Dr Al Jahuri added, “In this technique, low-dose 3D CT scans are combined with optical imaging with fixed cameras, which results in the creation of a three-dimensional display of augmented reality of the patient’s internal anatomy, and this combined image acts as an exploration device that guides the surgeon in fixing the vertebrae with surgical screws in place.

“This helps us conduct surgery that is precisely defined and reduces the chance of injury to nearby sensitive tissues such as nerves and blood vessels,” he went on to say. ““Following this approach leads to increased clinical accuracy as the surgeon can be sure that the surgical instruments are in the right place before the operation and this helps the doctor reduce surgical risks, medical errors, pain, and blood loss, which leads to faster patient recovery and reduces their in the hospital, as well as cuts down on subsequent medical complications and costs.”

The technology is expected to have a significant role in enabling surgeons to perform complex operations, especially in the field of spinal curvature. Thanks to the ability to adapt augmented reality to the needs of the surgeon, doctors can focus on what is happening right in front of them.

The tech also offers surgeons direct x-ray vision into anatomical areas that are not visible to the naked eye, which makes this new technology a mainstay in the future of surgery.

Ali bin Hamdan Al Ghafri, the patient who underwent the operation, said, “I was suffering from spinal pain, and after continuous review at the Armed Forces Hospital and the necessary medical examinations, the results showed friction in some of the vertebrae of the spine, and it was decided to perform a surgery using augmented reality technology.”

“I was briefed about the details and stages of this procedure from the doctors concerned with the operation, and what encouraged me to agree to perform the procedure was the presence of an Omani medical team of high reputation and efficiency, as well as what the Armed Forces Hospital possesses in terms of advanced modern devices in the field of spine surgery.”

Hanan bint Saif, who oversees the operating theatres at the hospital, said, “Spinal operations are complex procedures that require the surgeon to be careful when dealing with the nerves and vessels connected to the spinal cord.”

“Augmented reality technology is one of the latest technologies in spine surgery,” she added. “Globally, many doctors have been able to perform this surgery using this technique.”

The new service is offered by Ford’s Technical Assistance Center (TAC), a centralized diagnostic troubleshooting team that provides support to all Ford and Lincoln dealerships’ technicians who diagnose and repair customer vehicles.  Dealer technicians can initially reach out to TAC specialists via a web-based portal or even on a phone.  With the new See What I See program, TAC specialists can now start a remote AR session using TeamViewer Frontline through a pair of onsite RealWear smart glasses to share, in real time, exactly what the repair technician is looking at.  TAC specialists can add on-screen annotations and additional documentation directly in the line of sight of the repair technicians, as well as zoom in, share their screen, record the session and even turn on flashlights remotely.

“My team diagnoses some of the most complex and complicated vehicle issues,” says Bryan Jenkins, TAC powertrain operations manager.  “I would frequently hear my team say that if they could only see what that technician is talking about, or what the technician is doing or how they’re completing a test, then they could solve the problem more accurately.  A picture is worth 1000 words, but sometimes that still wasn’t quite enough, and we needed a way to see something live and in action.  And that’s what really kicked this whole program off.”

Ford’s See What I See program is an additional layer of support that is already used by more than 400 dealers in the U.S., Mexico, South Africa, Thailand, Australia, New Zealand and the U.K.  Currently Ford is promoting the new program to its full network of 3,100 U.S. based dealers, with a positive response. “Feedback from the dealers has been really good,” says Jenkins.  “From the dealer technician perspective, they just turn on their smart glasses and accept an incoming call, then it is like my specialists are there looking over their shoulder to help resolve the problem.”

“We are very excited to add Ford to our growing list of forward-thinking customers that are leveraging AR solutions to improve business processes,” says Patty Nagle, president of TeamViewer Americas.  “The majority of workers globally do not sit in front of a desk.  Our goal is to enable those frontline workers with AR guided solutions to enable them to do their jobs better by digitalizing and streamlining processes.”

While many AR companies are focused on building AR products, HP is making an interesting move in using the technology as an add-on to improve an existing line of its business. The company’s newly announced xRServices program promises to deliver remote AR support for its industrial printer customers.

The program employs Microsoft’s HoloLens 2 headset, which HP’s customers can use to access AR training and live guided instructions to fix issues that arrive with complex, commercial scale printers.

HP is pitching the solution as a way to allow even untrained individuals to fix issues with the help of a specialist on the other end who can guide them step-by-step through troubleshooting and repairs with AR instruction. Further the company says the service can be used to provide AR training for various workflows and issues that may arise with the company’s industrial printers.

HP hasn’t clearly detailed exactly what software it’s running on HoloLens to facilitate xRServices, but it seems likely that it is leveraging Microsoft’s Dynamics 365 Remote Assist platform which includes many of the AR functions that HP showcased in its xRServices concept video—like augmented annotation, document visualization, and video chatting through the headset.

The Bazeley Pilot Facility at the Parkville site in Melbourne, has been trying out Apprentice IO, an intelligent batch execution system that includes augmented reality. CSL, the world’s third largest biotech company, uses pilot plants to test manufacturing processes on a small scale. The company specializes in rare and serious diseases as well as influenza prevention.

Learn about a CSL Behring pilot plant in Illinois.

Working with Apprentice IO in Australia (AREA member) means a near-complete reimagining of core operations, including product development, manufacturing and supply chain solutions, said Sharon Orr, CSL’s Manager, Innovation and Technical Operations, Pilot Scale Operations. As part of the six-month test, the team is exploring alternatives to paper-based batch records, standard operating procedures and work instructions. The exercise has fundamentally changed the way operators of the system think about and approach instructions from the outset, Orr said.

When integrated with lab facilities, the augmented reality headset and linked iPad, can provide-on-the spot feedback, process directives and problem-solving techniques in real time. Paper records require a four-eyes approach for calculations, checking raw material information and weighs, she said. The experimental platform replaces manual cross-checking methods with automated formulas, ranges and barcoding. If results don’t match, then the system flags it to the operator saving time and ensuring compliance.

“Once a procedure has been augmented and approved, I can use this technology to perform a process ‘hands-free’ without having to worry about cumbersome paper-based data collection and manual checking,” Orr said.

Approved standard operating procedures can be accessed at the click of a button, Pilot Scale Operations scientist Hugh Harris said. With the new manufacturing software and augmented reality capabilities, colleagues at different sites would also be able to share data and see what’s happening in real time. CSL may bring the software system to other pilot facilities in Australia and the United States.

“Watching the team explore this new technology at the forefront of next generation manufacturing has been truly inspiring,” said Matthias Zimmermann, CSL’s Executive Director of Bioprocess Development. “They have shown a willingness to embrace the technology and incorporate it within our already existing processes. They have also worked with the software program designers to inform the next version, in some ways advancing this technology together.”

Geopogo is a California-based 3D design software company that is working to transform the design and construction process. The company’s software allows architects and designers to create renderings and a virtual reality (VR) or augmented reality experience in minutes by importing existing CAD models or building directly with the Geopogo 3D creator tool.

Now, with Geopogo’s software on Magic Leap’s AR headset platform, the interaction of digital content with the physical world will help to bring architectural designs to life, according to the companies. “This is a phenomenal opportunity to make architectural design understandable and accessible to project clients, city officials, and the general public,” said Geopogo’s Creative Director, Michael Hoppe.

According to Magic Leap, the American Institute of Architects, San Francisco (AIASF) utilized the partnership’s technology as part of its ‘Shape Your City’ campaign, an ongoing fundraising effort to build its new headquarters in the Bay Area’s new Center for Architecture + Design. The organization also sought to fund expanded architecture-focused tours, exhibitions, educational programs, and events for people of all ages.

As a result, AIASF hosted on-site building tours to build excitement and engagement for the project from the architectural community and the public, and offered tour participants a 3D virtual model of the future Center. The integration of AR technology during the building tours allowed for a more interactive, transparent, immersive, and exciting way to visualize what the space will look like, even before construction has started.

“The power of the AR experience succeeded in inspiring donors to contribute much-needed construction funding for the project, as hoped for by the non-profit organizations. We were especially happy to see how the AR experience brought so much delight to the faces of the non-profit Board, the organization members, and members of the larger community,” said Dave Alpert, Geopogo CEO and Cofounder. 

“The AR model has allowed our project partners, Board members, potential donors, and community to experience the future Center first-hand and visualize the positive impact it will have on future generations,” agreed AIASF Executive Director, Stacy Williams.

For more information on Geopogo and its augmented reality solutions for the architecture and design industry, click here. For more information on Magic Leap and its AR hardware solutions, click here.

The Challenge

With seven factories worldwide, Hirschmann Automotive needed a more cost-effective and time-efficient knowledge-transfer approach to maintaining and repairing equipment than flying experts around the world.

“If something isn’t working properly at one of our plants, technicians have to call our headquarters in Austria. And even then, they might not be able to solve the problem. Then it becomes an issue of flying someone around the world to assess the problem in person”

That’s when Fliri and his team looked at virtual and augmented reality solutions. Unfortunately, most devices were too delicate for the production plant environment — until Fliri discovered the RealWear HMT-1.

The Solution

Deploying RealWear running Cisco Webex Expert on Demand allowed Hirschmann Automotive to streamline collaboration and reduce equipment downtime.

The Results

• Reduced travel needs and costs
• Improved maintenance and repair response
• Streamlined information accessibility and collaboration
• Increased first-time fix rates
• Shortened first-time resolution time

Hands-Free Use Case

• Remote mentoring

Readers can download the case study for free on RealWear’s website