Stereoscopic vision has long been considered the best type of visualization in terms of matching physical and simulated realities. While stereoscopic vision is the goal, producing a perfect 3D visualization and registration of AR assets is difficult using current technologies. Addressing the shortcomings of current AR displays will be prohibitively high and present a financial barrier to AR adoption in enterprises.

Leveraging advancements in Digital Signal Processing (DSP) and audiology, a new class of devices are emerging. Spatially-aware audio transducers can help determine the exact position and posture/pose of the wearer as well as generate a simulated sound field that matches the physical environment. Such systems could be combined with existing vision-centric displays for high fidelity enterprise AR experiences.

The scope of this topic includes measurement of the spatial audio technology resource requirements and impacts of combining visual cues with spatial audio on user performance. Comparative studies of human cognitive performance aided by varying blends of spatial technology ranging from “audio-only” to “video-only” and various combinations of both are also in scope.

Stakeholders

AR experience designers, developers of integrated sensor and world capture components, human factors researchers

Possible Methodologies

This research topic will require development of visual and audio AR experiences to be produced in a highly controlled laboratory environment within which a series of experiments can be conducted and reproduced. Studies will compare spatial audio requirements to vision-only AR experiences on the basis of accuracy, speed, battery life, bandwidth requirements, processor performance, wearer comfort and pricing. In addition to user perception assessments through surveys and interviews, methods could be expanded to include time-motion studies using standardized, public and well-documented processes typical of industry verticals, use cases and horizontal use case categories.

Research Program

This topic is at the intersection of both 3D visualization and 3D audio. The methodologies and tools developed for this research could be used in the study of perception, presence, and lead to new guidelines for AR developers and manufacturers of HMDs for enterprise AR.

Miscellaneous Notes

In 2016, the Sound of Vision consortium, which focuses on the construction of a new prototype electronic travel aids for the blind published a report about audio-assisted vision. A peer-reviewed article presenting a novel technique for reproducing coherent audio visual images for multiple users, only wearing 3D glasses and without utilizing head tracking was published in 2011 in the Journal of The Audio Engineering Society.

Keywords

Spatial audio, effectiveness, spatial vision, 3D audio, perception, audio signal processing, acoustic waves, active noise control

Research Agenda Categories

Technology, End User and User Experience, Displays

Expected Impact Timeframe

Medium

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Kim, H., Remaggi, L., Jackson, P. J. B., & Hilton, A. (2019). Immersive Spatial Audio Reproduction for VR/AR Using Room Acoustic Modelling from 360° Images. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
• Heller, F., & Schöning, J. (2018). NavigaTone. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
• Schmalstieg, D. (2018). VIS Keynote Address : When Visualization Met Augmented Reality. 2018 IEEE Conference on Visual Analytics Science and Technology (VAST).
• Erkut, C., Holfelt, J., & Serafin, S. (2018). Mobile AR In and Out: Towards Delay-Based Modeling of Acoustic Scenes. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
• Abdelrazeq, A., Kohlschein, C., & Hees, F. (2019). A Cloud Based Augmented Reality Framework – Enabling User-Centered Interactive Systems Development. Advances in Intelligent Systems and Computing, 417-422.

More publications can be explored using the AREA FindAR research tool.

Author

Peter Orban, Christine Perey

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

In terms of collaborative robotics, the widespread adoption of robots in historically manual manufacturing environments (which are subject to high product turnover, short production runs, and high variability in equipment configurations) is limited by the robots’ inability to effectively and safely integrate and interact with the existing human labor. Instead, so-called collaborative robots are relegated to secluded operations with minimal contact with the workforce. The robots’ inability to communicate with, understand the intention of, and establish a mutual understanding of the environment and situation with human coworkers decreases the robots’ usefulness in collaborative teams consisting of both robots and people. This limitation is driven by both the absence of tools and protocols needed for effectively describing and measuring human-robot interactions, an incomplete collection of metrics for assessing human-robot teaming performance, and insufficient protocols for enabling more intuitive interfacing with robotic tools. These challenges are compoudned when augmented reality technologies are used at the interface between the robotics and human workers.

This research topic focuses on providing the methods, protocols, and metrics necessary to evaluate the interactive and teaming capabilities of robot systems. It uses a task-driven decomposition of manufacturing processes to assess and assure the safety and effectiveness of human-robot collaborative teams.

Stakeholders

Manufacturers will benefit from the products generated as a result from this research project. Robotics providers can also benefit in that standard testing protocols for human-robot interaction will generate new sales tactics. End users will benefit in that the end state will be much safer in complex manufacturing environments.

Possible Methodologies

This collection of methods, protocols, and metrics will enable integrators and end-users to maximize the effectiveness and efficiency of collaborative human-robot teams in production processes, impacting both large-scale companies designing and repurposing hybrid manufacturing workflows, and smaller companies looking to begin introducing automated tools into manual processes.

Research Program

This research topic mirrors an existing project at NIST. Inspiration can be driven from the existing work generated by that team. Furthermore, IEEE is a leader in curating academic work in this area. Refer to IEEE RAS for related publication venues, including IEEE CASE, IEEE ICRA, and IEEE IROS.

Miscellaneous Notes

This topic requires significant hardware, middleware, and software integration. One open source framework is ROS-Industrial

Keywords

Robotics, human-robot interaction, human-computer interaction, remote monitoring, remote control, collaborative robots, autonomous agents, communication, computer vision, control systems, cooperative systems, grippers, human factors, human-robot interaction, industrial robots, industry 4.0, intelligent robots, multi-robot systems, occupational safety, robotics, safety

Research Agenda Categories

Standards, Technology, End User and User Experience

Expected Impact Timeframe

Long

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Muhammad, F., Hassan, A., Cleaver, A., & Sinapov, J. (2019). Creating a Shared Reality with Robots. 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). </li>
• Jana Jost; Thomas Kirks; Preity Gupta; Dennis Lünsch; Jonas Stenzel (2018). Safe Human-Robot-Interaction in Highly Flexible Warehouses using Augmented Reality and Heterogenous Fleet Management System. 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR).
• Bolano, G., Juelg, C., Roennau, A., & Dillmann, R. (2019). Transparent Robot Behavior Using Augmented Reality in Close Human-Robot Interaction. 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).
• Whitsell, B., & Artemiadis, P. (2017). Physical Human-Robot Interaction (pHRI) in 6 DOF With Asymmetric Cooperation. IEEE Access, 5, 10834-10845.
• Xue, C., Qiao, Y., & Murray, N. (2020). Enabling Human-Robot-Interaction for Remote Robotic Operation via Augmented Reality. 2020 IEEE 21st International Symposium on “A World of Wireless, Mobile and Multimedia Networks” (WoWMoM).

More publications can be explored using the AREA FindAR research tool.

Author

Bill Bernstein

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

Significant AR experience development effort and time is dedicated to linking domain-specific data to AR engines and presentation systems. While many industries invest in AR scene development, the need is especially high in the manufacturing industry. Product Lifecycle Management (PLM) software systems curate design, manufacturing, and sustainment information related to a product system (ideally) throughout its entire lifecycle.

Leveraging such data across PLM systems for industrial AR has mainly become a platform-specific exercise. Commercial PLM software vendors provide industrial AR solutions that leverage their application programming interfaces (APIs) to help establish AR scenes. However, most large original equipment manufacturers (OEMs) operate in complex, global, and heavily distributed supply chains. In other words, in many cases, OEMs subscribe to every major commercial PLM software platform to handle the variety of data representations provided by their suppliers. This is especially important if the developers aim to truly create a digital twin of a product or production system.

This research topic will develop and test a standard data model that permits the vendor-neutral exchange of AR-critical data to produce updatable, sustainable, and maintainable AR scenes. One particular area of interest is understanding the proper handling of industrial animations. Though AR standards provide representations for presenting animations within AR scenes, for example, they do not directly relate to product system animations stored within PLM systems. Rather than relying on a particular PLM platform’s data representations, if such a data model was successfully developed, software developers would be able to exchange data across PLM software and to platform-agnostic AR engines more readily.

Stakeholders

OEM manufacturers, integrated solution and software developers, CAD/CAM providers, engineering design teams, PLM software publishers.

Possible Methodologies

This research topics could focus on testing existing PLM platforms and their AR-related competencies. Reporting on gaps bewteen the interfaces across related software tools will be a strong contribution. Existing standard data representations provide a strong starting point for investigation. Focusing on dealing with sustainment for digital twin models would cover lots of the use cases.

Research Program

OEMs with complex and heavily distributed supply chains should be a center point for the project. Program managers and technical advisors in such organizations understand the issue of cumbersome technical data packages. This research topic significantly overlaps with the Digital Model Interoperability research topic. [BB: issue with sentence grammar] There are distinct in that this research topic is specific to the manufacturing industry. The Digital Model Interoperability research topic relates more generally to 3D asset translation and can be applied to other domains, such as construction.

Miscellaneous Notes

There exist resources to help position research efforts. For example, the CAx Implementor Forum provides a number of test cases. The Khronos Group is continously updating glTF, the latest de-facto standard for lightweight model presentation. ASME Y14 is a working group that focuses on the standard presentaiton of GD&T annotations.

Keywords

Standards, data interoperability, digital twin, product lifecycle management, platform-agnostic AR solutions, application programming interfaces, distributed supply chains, manufacturing, sustainment, industrial Internet of Things, IIoT, ontologies (artificial intelligence), cost engineering, knowledge engineering, information management, project management, supply chains, team working, asset management, metadata, application programming interfaces, open source software, design engineering, systems engineering, cad/cam, quality management, supply chain management, manufacturing industries, product life cycle management, standardization, interoperability

Research Agenda Categories

Standards, Technology, Industries

Expected Impact Timeframe

Near

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Eckertz, D., Berssenbrügge, J., Anacker, H., & Dumitrescu, R. (2019). Work-in-Progress: Enhancing Collaboration Using Augmented Reality Design Reviews for Product Validation on the Example of Additive Manufacturing. Cyber-Physical Systems and Digital Twins, 244-254.
• Speicher, M., Hall, B. D., Yu, A., Zhang, B., Zhang, H., Nebeling, J., & Nebeling, M. (2018). XD-AR. Proceedings of the ACM on Human-Computer Interaction, 2(EICS), 1-24.
• Tobias Küster; Nils Masuch; Johannes Fahndrich; Gudrun Tschirner-Vinke; Jan Taschner; Markus Specker; Hendrik Iben. A Distributed Architecture for Modular and Dynamic Augmented Reality Processes. 2019 IEEE 17th International Conference on Industrial Informatics (INDIN).
• Bellalouna, F. (2020). Industrial Use Cases for Augmented Reality Application. 2020 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom).
• Zigart, T., & Schlund, S. (2020). Evaluation of Augmented Reality Technologies in Manufacturing – A Literature Review. Advances in Human Factors and Systems Interaction, 75-82.

More publications can be explored using the AREA FindAR research tool.

Author

Bill Bernstein

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

Creating AR experiences using existing authoring environments is a time-consuming process even for highly-trained developers. One of the greatest hurdles is the specification of real world features to which AR assets are attached/anchored. An alternative to manually defining anchors for AR assets is to adapt the outputs of real world 3D capture systems used in enterprises to be used as anchors for AR.

New interfaces and data encodings for 3D capture systems to make those systems directly accessible in an AR authoring pipeline will streamline or automate the preparation of AR experiences based in part on 3D capture of enterprise environments. However, there are many different 3D capture systems. This research topic focuses on development of guidelines and/or standards that will define the formats for 3D real world mapping which can be used by commercial AR authoring platform developers and publishers.

Stakeholders

This research is relevant to AR experience developers, AR managers, AR authoring platform publishing companies, AR service providers,

Possible Methodologies

Requirements for this approach for authoring AR experiences will be compared with existing and new standards and/or extensions of existing standards to automate AR authoring. The gaps and requirements will more easily be provided to appropriate standards development organizations for future work. If and when new interfaces and standards are published, enterprise AR authoring software vendors will be able to use these to streamline the AR authoring pipelines in industry.

Research Program

This topic could be combined with other topics to increase efficiencies in AR asset and experience authoring, management and delivery to reduce time and costs of integration with existing authoring and data management systems and platforms.

Miscellaneous Notes

This topic was submitted for 8th and 9th AREA research projects and received high support.

Keywords

3D world capture, depth sensing, liDAR, AR experience authoring, AR assets, anchoring, world mapping, SLAM, simultaneous localization and mapping, standards, data preparation, object-oriented programming, open systems, authoring systems, shape recognition, feature extraction, 3d modeling

Research Agenda Categories

Standards, Technology

Expected Impact Timeframe

Near

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Cavallo, M., & Forbes, A. G. (2019). CAVE-AR: A VR Authoring System to Interactively Design, Simulate, and Debug Multi-user AR Experiences. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
• Speicher, M., Hall, B. D., Yu, A., Zhang, B., Zhang, H., Nebeling, J., & Nebeling, M. (2018). XD-AR. Proceedings of the ACM on Human-Computer Interaction, 2(EICS), 1-24.
• Babu, N. S. C. (2017). Keynote 1: Internet of Things(IoT) and augmented reality for e-learning. 2017 5th National Conference on E-Learning & E-Learning Technologies (ELELTECH).
• Blattgerste, J., Renner, P., & Pfeiffer, T. (2019). Authorable augmented reality instructions for assistance and training in work environments. Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia.
• Zigart, T., & Schlund, S. (2020). Evaluation of Augmented Reality Technologies in Manufacturing – A Literature Review. Advances in Human Factors and Systems Interaction, 75-82.

More publications can be explored using the AREA FindAR research tool.

Author

Christine Perey

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard

Many industries require that employees work in high risk environments. For this reason, employees are certified in advance of performing tasks and follow very strict protocols. When sensors on a device or during a work flow detect that a high risk or dangerous setting or task is imminent, there needs to be an automatically-displayed alert. Research that increases the reliability of automatic risk assessment based on situational awareness from employee-worn sensors, regardless of the technology provider, and produces a consistent and clear response regardless of the AR device model, mode or connection state would provide value to those working in high risk industries.

This research topic will extend the user experiences that are currently provided on smartphones in the workplace. It will also need to clarify when and how existing alert systems prioritize the messaging to users and test usability and benefit of having the alerts appear in AR view.

Stakeholders

Standards bodies, Safety officers, employees working in high risk jobs

Possible Methodologies

This topic requires consensus building by working with safety officers, and new approaches to create and codify use of AI (and train the AI) to detect high risk context and circumstances, and to immediately alert the AR user. The research could also study different types of alerts for different industries, their effectiveness and recommend protocols that all devices and AR-based systems could adopt.

Research Program

This topic or theme of research overlaps with the topic of visualizing alerts in a timely, intuitive and meaningful way. The outcomes of this research could also be applied in many industries. It could be combined with topics pertaining to safety and human factors.

Miscellaneous Notes

Standards for automatic alerts could also be applied in many industries where danger is minimal. It could be combined with topics pertaining to safety and human factors. Could be an offshoot of UL8400 or BSI IST/31.

Keywords

Safety, risk, automated alert, standards, situational awareness, sensors for risk detection, artificial intelligence, health risks, occupational risks, risk assessment, risk perception, accidents, safety-critical software, occupational health, occupational safety, safety, accident prevention, disaster prevention, electrical safety, health and safety, health hazards, safety devices, safety factor, safety systems, fault detection, monitoring, system monitoring

Research Agenda Categories

Standards, Technology

Expected Impact Timeframe

Near

Related Publications

Using the words in this topic description and Natural Language Processing analysis of publications in the AREA FindAR database, the references below have the highest number of matches with this topic:

• Lutz, R. R. (2018). Safe-AR: Reducing Risk While Augmenting Reality. 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE).
• Aromaa, S., Väätänen, A., Aaltonen, I., Goriachev, V., Helin, K., & Karjalainen, J. (2020). Awareness of the real-world environment when using augmented reality head-mounted display. Applied Ergonomics, 88, 103145.
• Ko, H., Oksana, B., Na, I. S., & Pan, S. B. (2018). Analysis an identification with ECG base in augmented-reality. Proceedings of the 2018 Conference on Research in Adaptive and Convergent Systems.
• Aromaa, S., Väätänen, A., Kaasinen, E., Uimonen, M., & Siltanen, S. (2018). Human Factors and Ergonomics Evaluation of a Tablet Based Augmented Reality System in Maintenance Work. Proceedings of the 22nd International Academic Mindtrek Conference.
• MITSUHASHI, K. (2018). Suggestion of the Booting System for Necessary Safety Check by Augmented Reality and Computer Graphics. 2018 9th International Conference on Awareness Science and Technology (iCAST).

More publications can be explored using the AREA FindAR research tool.

Author

Christine Perey

Last Published (yyyy-mm-dd)

2021-08-31

Go to Enterprise AR Research Topic Interactive Dashboard