An Augmented Reality Platform for an Oil and Gas Enterprise

  • An AR platform for real-time audio, video and chat collaboration
  • An Android application for AR glasses
  • End-to-end communication encryption
  • An in-built ticketing system
Expertise
Mobility
Collaboration Solutions
Verticals
Oil & Gas
Technologies
Java
Mobile

Iflexion helped an oil and gas enterprise overcome the pandemic-imposed traveling restrictions and cut equipment maintenance expenses by developing a robust and secure augmented reality platform for real-time audio and video communication between field workers and external experts.

  • WebRTC
  • Java
  • Kotlin
  • Node.js
  • Python
Calendar
4
months
Gears
11
people

Context

Our customer is one of the world's largest enterprises in the oil and gas industry with 20,000+ employees and 1,400+ customers in 70+ countries. 

The company has very expensive and sophisticated equipment that, in case of failure, they would repair in-house or by inviting external experts from abroad. In addition to high associated expenses that could reach up to $10,000 per one repair job, the restrictions imposed by the COVID-19 containment rules made it no longer possible to get direct help from foreign experts.  

To address this issue and improve the maintenance workflows, the customer wanted to apply advanced technologies in the form of an AR-enabled platform for collaboration between field technicians and remote experts.  

Their main requirements for the solution were: 

  • Compatibility with RealWear and Epson smart devices in use on the company’s premises.
  • Extensible architecture allowing for easy implementation of new features and changes in the future 
  • Stable and secure audio and video streaming in real-time
  • In-freeze mode support
  • A chat for exchanging messages, screenshots, and stickers
  • A local storage of video files and a backend database storage for audio files
  • End-to-end audio and video traffic encryption and the possibility of decryption in case of incident investigation

The company chose Iflexion owing to our expertise in augmented reality development, extensive knowledge of WebRTC technology, outstanding customer references, and a successful completion of a test task.

Solution

Iflexion developed an augmented reality platform for cooperation between on-site and remote oil and gas specialists. It allows the company’s field workers to connect with internal or external maintenance experts, share audio and video via their AR glasses, and receive step-by-step instructions regarding repair and maintenance of the company’s equipment.

User Roles

The platform offers three major user roles, each with an individual set of permissions and responsibilities: 

  1. Administrator/Operator (usually a shift engineer) can overview the platform operation, monitor system alerts, and create corresponding tickets. They can also decrypt and merge video and audio content transmitted from smart devices and save decrypted files to their desktop. 
  2. Field technicians can start and terminate audio and video communication with remote experts, take photos and videos, and record them to the device’s internal storage. 
  3. Remote experts can start and terminate audio and video communication with field technicians, send images and text messages, and share their computer screen to AR glasses.

Real-time Communication

To start a video session, the technician scans the QR code of the equipment in need of repair. After this, a remote expert receives an email with a link for joining the video conference. Before entering the call, the remote expert must pass the 2-factor identification via their mobile phone number. 

Iflexion’s team equipped the platform with the voice-first interface to free field workers’ hands. With speech commands, technicians can start remote broadcasting from their AR glasses, take high-resolution photos, and record HD and Full HD videos. 

Owing to high photo and video resolution, remote experts can view equipment in detail and accurately identify the problem. The platform also allows ‘freezing’ the virtual object so that technicians can better inspect it. The remote specialist’s instructions are broadcasted to the customer’s proprietary web server and displayed on the AR glasses screen in real time. 

Our developers also equipped the solution with chat functionality. In case internet connection is poor, the remote expert can send the field worker signal stickers, such as "right" if everything is done correctly or "stop" if the worker should stop what they are doing.

A Ticketing System

We integrated an end-to-end ticketing system into the platform, fully digitizing the remote assistance workflow. The feature provides the customer with a complete visibility into the issue resolution cycle, from equipment information verification to maintenance and repair cost assessment. The tickets also contain such details as: 

  • Skills and qualifications of internal or external experts
  • Who, when, and from whom obtained the corporate AR equipment
  • System support specifics
  • Consultation progress and results

Security

One of the major concerns for the customer was the security of their confidential corporate information passing through AR-enabled devices. Due to this, the project had stringent security requirements. 

To protect the confidential information shared during a video conference and prevent its covert distribution, all data channels are encrypted with certificates issued by the customer’s corporate center. In addition, each video session has a unique temporary access ID and a time-limited HTTPS link. 

At the end of each video call, the recordings are encrypted and stored in the device’s private data area and then saved to the SD card and the desktop. 

To increase process visibility, Iflexion’s software engineers have developed a Python-based utility that decrypts records, in case equipment is mishandled and an investigation is required. When the operator/administrator saves video and audio records to the computer, the utility automatically decrypts and processes the files. The utility also allows for merging several short video files into a single one and adding an audio track to the video file. 

Finally, to protect the customer’s information from rooting-caused exploits, we built an automated testing script for the AR app. If a field worker activates AR glasses and the device is identified as rooted, all data, including audio and video files, chat messages, and screenshots, together with the Android application are automatically deleted.

Technologies

The platform was developed for Epson (Android 4.0) and RealWear (Android 8.0) devices. To implement the core functionality — high-quality real-time video communication — Iflexion’s team utilized the WebRTC technology and Java. We also relied on the RxJava library to control multiple complex tasks in a thread. 

The platform consists of four modules: 

  • The Media server with full WebRTC support that enables sending, receiving, mixing, and transcoding media streams between AR glasses and desktop computers. 
  • The Web server built on Node.js that facilitates management of the media server and AR glasses, interaction with corporate microservices, and desktop access to the platform. 
  • The Web app developed with HTML5, CSS, and JavaScript allows users to record video and audio, generates media streams, and sends them to the media server via the SIP protocol, which then transfers it to the computer screen and headphones. 
  • The AR app for Android-based AR glasses records video from the AR glasses camera and audio from the microphone, generates media streams, and sends them to the media server, which transmits media to the AR glasses screen and headphones.

AR Application Architecture

We developed the AR app applying the Model-View-ViewModel (MVVM) architecture, which allowed us to separate business logic and data from the user interface and the presentation logic and thus ensure the software’s ease of maintenance.

We also set up two databases: Data Source keeps data received from the media and web servers, while Local Data Source stores audio and video files.

Results

Within four months, Iflexion developed an augmented reality remote collaboration platform for the oil and gas enterprise, which includes media and web servers, a web app, and an Android AR app for smart headsets. The solution facilitates high-quality real-time communication between field technicians and in-house or remote experts via video, audio, and chat. 

Fitted with robust features, the solution allowed the customer to: 

  • Improve production line performance
  • Reduce equipment downtime
  • Lower maintenance costs by up to $10K per job
  • Continuously address maintenance-related skill gaps 

After the solution release, Iflexion’s team moved on to provide ongoing maintenance and support of the AR platform, introducing smaller improvements to increase its efficiency for the customer and their end users.

Screenshots

WANT TO START A PROJECT?

It’s simple!

Attach file
Up to 5 attachments. File must be less than 5 MB.
By submitting this form I give my consent for Iflexion to process my personal data pursuant to Iflexion Privacy and Cookies Policy.