One Billion Points Streamed In Augmented Reality

Ships regularly undergo large-scale retrofits, but shipowners rarely have design data in digital form at hand. To design ship modifications, an engineering accuracy of the as-built ship geometry is required, which means each vessel must be 3D laser-scanned. ShipReality, a company specialised in AR/VR ship design automation and remote ops, synthesises these large ship laser scans with its CAD software to design directly in 3D, resulting in merged models of CAD in the as-built ship geometry point clouds.

“We want to speed up and optimise retrofit designs for 60,000 ships that require greenhouse gas emissions reduction, energy conversions & ballast water treatment system (BWTS) retrofits in the coming years”, said Georgios Bourtzos, CEO and co-founder of ShipReality. “A major challenge we faced designing directly in large point clouds was visualising entire vessels layered with resulting 3D designs for immersive design reviews on mobile XR devices like Oculus Quest 2 and HoloLens 2.”

Exploring the use of point clouds in AR

Point clouds are precise models of real environments based on 3D laser or photogrammetry scanning. Objects and space are represented in the form of “points”. Millions of such points combined formulate a point cloud scan. The scan is then imported into a 3D modeling platform with the purpose of creating an as-built model. Common CAD software used for ship designs, although incorporating 3D laser scans, still rely on 2D projections and screens to visualise and design in 3D. This often results in incompatibilities with the existing ship geometry, which are only realised during installation, creating substantial delays and high additional costs.

Visualisation in 3D is a key issue to address these problems. However, visualising large point clouds requires substantial CPU and graphics power. The performance requirements are simply too high for rendering it locally on a mobile XR device. It would cause an extremely low frame rate and even software crashes. Dealing with large datasets, ShipReality had to find a solution that could surpass the limited memory, CPU and GPU resources of mobile devices.

XR streaming solution that supports point cloud

XR streaming technology outsources the rendering process to a local server or the cloud. But not every solution on the market supports ultra large data or point clouds. “We integrated the ISAR SDK into our solution to stream large 3D laser scans merged with CAD retrofit designs to a HoloLens 2,” said Mr Bourtzos. ISAR (Interactive Streaming for Augmented Reality) is a unique remote rendering software component that allows to stream entire augmented and virtual reality applications in real time. “The simplicity and ease of integration of the software development kit worked seamlessly with our large models.”

Integrating ISAR into their engine and software, ShipReality was able to visualise a massive model containing more than one billion points. Layered on top was also BWTS CAD design data created by the company’s ShipMR-design software with additional five million polygons. To compare, a mobile XR device could only render about one and half million polygons locally. As the remote rendering server ShipReality used a moderate gaming laptop and the local WiFi, broadcasting on 2.4GHz band. More performance and bandwidth enable even greater visualisations.

Next level immersive experiences

“ISAR has amazing potential for AR/VR visualisation of massive digital twins and real-time monitoring of projected complex 3D designs merged with as-built environments in shipping and other industries,” said Mr Bourtzos. ShipReality is now able to:

• visualise ship models that are only suited for high-performance processing
• capture large assets 1:1 and integrate complex 3D CAD designs/data
• visualise detailed models for spatial analytics in augmented reality

“ISAR can save us a lot of time and resources because we can directly use point clouds in mobile AR: some pre-processing steps can be avoided.”

Data availability, accuracy, density, and size of 3D point clouds are also forecast to vastly increase within the next years. “To realise the full potential of immersive point cloud experiences, streaming will play a major role,” concluded Georgios Bourtzos.