How One Airline Is Using AR to Improve Operations

Safety 2.0

One of the first processes to benefit from CSA’s embrace of the augmented reality is the safety inspection. (While the cause of the recent crash of a Boeing 737 operated by China Eastern Airlines is still being investigated, the incident underscores the importance of using any and all safety inspection tools available.)

After every landing of any passenger aircraft anywhere in the world, a maintenance, repair, and operations (MRO) engineer must perform a thorough aircraft inspection. An inspection on a Boeing 737 typically takes more than 100 steps, and an Airbus 320 takes more than 200 steps. This is a basic but crucial part of airline management, a pressing task that front-line MRO engineers must perform every day, often a number of times. CSA is no exception. As the world’s third largest airline, the airline repeats this procedure over 2,500 times a day, a task that takes CSA’s MRO engineers about 1,000 worker-hours.

In most of the world’s airports, engineers check off each inspection item on a bulky, paper-based job card, a block of 20+ sheets of paper that they have to clutch throughout the entire task. Until recently, CSA’s MRO engineers worked this way too, performing this work while juggling paper, pens, walkie-talkies, and the job card. But now, at 22 airports the CSA flies out of, most of the information, record-keeping, and communications tools are integrated into a single AR display. This display puts a whole range of resources at the engineers’ service – not only text, but images, videos, graphs, and voice, in any combination that is helpful to the engineers.

While the AR glasses are expected to shave 6 percent off those 1,000 daily hours, we have found in our research on the integration of this technology at CSA that the advantages of the AR glasses go far beyond the labor dividend. They aren’t just a new way to get information – they’re a whole new way of working.

CSA’s AR glasses allow engineers to edit and reorganize their job list, change the information they see, and how they want it shown. Their displays can be adjusted by aircraft, season, and even individual preference. They offer the engineers step-by-step multimedia support and immersive experiences during the execution of the tasks, including AI object recognition and collaboration with a remote expert.

“Combined with some [artificial intelligence], the AR glasses can really make our job a lot easier,” one MRO engineer said. “I can now point my fingers to a place, for example, a lubricating oil cap, and it automatically recognizes the object or the key parts and tells me that it’s open but should be closed. It also can show me, in a picture or a short video, how the object looked in normal condition or in its last service.” When the task is done, engineers can even sign off using either voice or even gesture, if it’s too noisy on the tarmac to use a voice command.

Rather than lug manuals big as unabridged dictionaries around or spend valuable time walking to an office to consult one, engineers can instantly access the information they need via the glasses. “I no longer have to go and look for the maintenance manual, which could take an hour walking back and forth. The manual is now coming to me, in front of my eyes!” one engineer told us. The AR glasses even make it possible for experts to advise mechanics on the tarmac in real time and supply them with pictures, videos, voice advice, and graphs.

The glasses also encourage more standardized performance. “It knows where I am in the process and points me to where I need to go next. Everybody is following the same process in the same order,” explained another engineer.

Welcome to the Augmented Operations

Wide-awake engineers, better compliance, a visual diary of the life of every component, and ultimately, safer flights are all benefits of this single pilot project in the 850-aircraft airline. The AR glasses optimize performance not only by bringing more knowledge closer to the machinery but by keeping MRO eyes on the prize. Like most earlier forms of digitalization, the CSA’s experience suggests that augmented operations are less likely to supplant people than to augment their capabilities — a win for companies, employees, and travelers.

Today, the CSA’s first augmented operational system is still a work in progress, not so much in its ability to transfer data to or from the individual — although that presents challenges – as in adapting the technology to meet the capacities of human cognition. The AR smart glasses need to comply to the industry’s safety standards, as well as meeting important objectives for privacy, comfort, display, connectivity, ergonomics, battery lifetime, noise reduction, multi-media interactivity, immersive experience combined with transparency, required infrastructure (5G, edge computing), and a knowledge graph that can provide deeper AI-enabled support.

Only the beginning

And that’s just one application in one industry — imagine the many other ways the technology might be used. Already, thousands of companies around the world are experimenting with various aspects of AR technologies. And we believe this number will rise dramatically once we understand more about the best ways to manage the user interface on all those smart glasses, and the awareness of this new and highly adaptable technology grows. It’s not unlike the moment when something called a website appeared on our desktops or a decade later, when it became clear that apps were “the killer app” of the smart phone.

When the enterprise use of AR technologies has its own Netscape moment, we believe we will see many industries see the dawn of a massive new opportunity. Airlines, for instance, will be able to understand their cost structure in much more detail than they do now, down to the part. Ultimately, this cognitive shift could change the balance of power within the airline business, away from sales and the front office toward the back office and the maintenance hangar (particularly as the carbon footprint becomes more integrated into the price).

And this is only the beginning. As the CSA project has demonstrated, virtualization has no limits. Any person or object in airline operations, from mechanics to the airplanes or the entire airport — can be virtualized, given enough data and enough modeling. By creating a virtual representation of a physical object, plus an ongoing stream of new information about its status, digital twins of physical objects and even people can give airlines unprecedented ability to see how something is performing right now and simulate or predict how it might perform in the future.

CSA’s success suggests that AR is finally becoming part of our working reality. But there are still many unanswered questions. In our work for CSA, for instance, we have been posing many questions about how to bring the best out of people. When do people need reminders? What are the signs that their attention is starting to flag? What’s the most efficient way for mechanics to communicate through their glasses to an expert, who can walk them through a complex repair? At the moment, the questions keep multiplying but, fortunately, so do the answers.