By Marlo Brooke & Scott Toppel
This paper presents best practices for implementing Augmented Reality in ultrasonic and electrical Non-Destructive Testing for aircraft. These best practices will enable aerospace supply chain stakeholders to establish AR within their Industry 4.0 and digital transformation roadmap.
There are different ways to test aircraft safety. One way is by dropping the aircraft from many thousands of feet, which obviously destroys the aircraft. Another way is through Non-Destructive Testing (NDT). Also referred to as non-destructive evaluation (NDE) or non-destructive inspection (NDI), NDT is an aptly named testing method that does not destroy equipment. While NDT doesn’t replace drop and flutter testing while the aircraft is being developed, NDT does enable visibility to structural degradation before major critical damage has occurred, so aircraft fatigue can be addressed more cost-effectively.
NDT allows inspectors to detect minute cracks – not visible to the human eye – within the aircraft structure, through ultrasonic sound and electrical (eddy) current. Ultrasound and eddy current allows us to identify microscopic cracks through an alteration of the electromagnetic or ultrasonic field, so we can see identify the precise location and severity of the structural risk.
By using the NDT inspection technique to initiate proactive aircraft maintenance, both catastrophic accidents and the high cost of more expensive maintenance can be avoided. Certainly, extending the usability of a $30-90M asset has a significant ROI year after year.
A significant challenge of eddy current and ultrasonic testing is a high learning curve. Because of its complex nature, it can take upwards of two years to train new inspectors on these methodologies. Learning the physics of these techniques is not straightforward.
The second barrier to successful NDT is that even with the best of inspection equipment, inspections results are often subjective because NDT process itself is highly manual and laborious.
These barriers to success are significantly alleviated with Augmented Reality (AR). By enabling the inspector to visualize in 3D, AR significantly reduces the learning curve and improves accuracy for many types of ultrasonic and eddy current inspection. AR also enforces standards and parameters, which in turn removes subjectivity from the analysis.
AR for NDT Best Practice #1: Implement a Single System for Training and Inspection
The first step to success is designing the AR application in dual use mode. Dual use means that you canuse one solution for both inspection training, and as a real-time job aid during the inspection itself.
Open Space or Offsite Mode
If the inspector does not have equipment to train on, they will use AR in open space, also known as offsite mode. In open space, the aircraft inspector can pre-train, train, and re-train on the processes and equipment as many times as needed, anywhere (even at home). This is good “reps and sets” type training to embed knowledge into the inspector prior to even touching the device.
Traditional NDT classroom training involves PowerPoint slides and other 2D training and reading materials. A game-changing benefit of AR is its ability to teach complex physics concepts, through 3D representation in the real world. The technician not only needs to learn the concepts, but also how to locate and identify invisible cracks in a large structure using invisible sound and eddy current.
AR makes the unseen visible. It is substantially easier to understand and apply the physics of sound and electrical current when in real time and space, as opposed to understanding these concepts from books and flat images. AR provides inspectors an intuitive visualization of electrical current and/or ultrasonic sound while inspecting the aircraft – making it immensely easier to identify degradation.
The second constraint of traditional NDT training is classroom equipment. During training, the NDT trainee uses an inspection unit along with a physical block that mimics an aircraft structure – this block has a finite number and type of defects built into it. When the trainee places the inspection probe into a crack, different shapes are displayed that allow the inspector to determine information about the defect. The trainee learns about both the inspection tool and the way to assess the crack. The issue with learning this way is that the block is limited in its number of defects, and changing the physical block is time intensive.
AR virtualizes this effort, which in turn expands the training experience and comprehension. For example, using ARinspect™, the trainee can have a virtualized system with an unlimited number of defects to test and practice against, without needing a physical training tool. This supports training anywhere and enables a broader and more comprehensive interactive training by using a digital representation of the equipment and error. In AR offsite mode, the inspector can place the inspection probe (or a virtual representative) into a virtual hole with a virtual defect.
The optimal AR training in offsite mode incorporates individualized scaffolded learning techniques – or learning hints – that enables a progression of learning assistance depending on the trainee’s needs. Gamification is incorporated to improve memory retention. It can involve random algorithms to change location of crack to make it more game based, interactive, and even competitive in a team environment.
AR furthermore enables real-time and automated user-specific performance assessment, as well as inspection results. This enables twofold benefit. Firstly, a trainer or senior expert can monitor the technician’s performance both in training and during the actual inspection itself. Secondly, it saves time and potential data entry errors by automating the results electronically, replacing the current method of the inspector needing to stop frequently and record their findings.
Real Environment or Onsite Mode
Because AR is so effective as a job aid, many organizations will begin to use AR in the field, even prior to establishing AR within its formal training curriculum. With digital overlays on a wearable device that provide visual an audio cues, AR is exceptionally powerful in reducing time on task, in addition to many other benefits described herein. A NAVAIR thesis on aircraft maintenance using AR showed a 100% reduction of time on task. (“Effects of Augmented Reality Based Object Illumination on Human Performance”, Matthew D. Stone, NAVAIR Public Release 2020-328.)
This powerful second part of ARinspect™ dual use mode is called real environment, or onsite mode. In real environment mode, the aircraft inspector is working with the inspection equipment and aircraft, performing the inspection (or training) while viewing AR overlays that provide additional information within sight, when where and how the content is needed. All of the advantages described in training mode are now set to use during real world inspection, which enables a highly automated and elegant inspection, as well as automation of manual inspection tasks.
Designing one AR solution for dual use mode is a best practice for two reasons. Firstly, the inspector can train using the same AR device that is used during inspection – which further reduces any learning curve associated with implementing a new training method. It goes back to the adage, “train like you maintain.” Secondly, from an implementation and sustainment perspective it is much more cost effective to support one solution for both. Features may be slightly different between uses for training versus on-job, but these functions can be programmed to be turned on and off. For example, gamification is not necessary in the field, but data capture of defects is extremely important.
Inspection equipment calibration is critical to the NDT process, since it is the source of defect detection. Just as AR empowers the technician by overlaying concepts and information on the aircraft, AR should be used for learning and calibrating the inspection equipment itself.
AR for NDT Best Practice #2: Reduce Inspection Subjectivity
A best practice when designing AR is to incorporate Artificial Intelligence (AI) for specific parts of the AR application. It is important to understand how AI produces the best results during NDT. To begin, NDT inspectors are highly trained, certified and proficient in their field of expertise. Their career requires utmost integrity and expertise that is honed over many years. The best AR applications support the inspector – they cannot remove the critical human decision processes that provide substantial benefits to the aviation industry and air travel safety as a whole.
Unlike some fantastical Hollywood depictions of technology somehow “replacing” complex human thought and perception, AR should not be considered a replacement of experienced inspectors. Instead – as the term augmented reality suggests – it can only augment the technician so they can be better at their craft, and ultimately ensure safe aircraft for all of us. In the paper “Intelligence Automation in Nondestructive Evaluation” (AIP Conference Proceedings 2102, 020028: 2019: Air Force Research Laboratory, John Aldrin, Eric Lindgren, David Forsyth) the authors flip the acronym, and dispel the fantasy about Artificial Intelligence replacing people. Very similar to AI, and yet fundamentally different in approach, they describe the term Intelligence Automation (IA), which “refers to the effective use of information technology to enhance human intelligence.” Here the authors substantiate and clarify the paradigm that AI is to support the human being in making better decisions, which optimizes the outcome for all.
With this approach in mind, AR best supports NDT when it is combined with AI for two functions: Auto Detect, and Assessment. This functionality must be designed from the viewpoint of how to enhance the user experience and make the inspection process easier, faster, and more effective.
Auto Detect
AR Auto Detect uses AI to provide automatic QA and recording of inspection results in the field. This saves time for the inspector, while increasing accuracy. During eddy current bolt hole inspection, for example, AR can automatically identify and color code each of the many bolt holes that need to be inspected, ensuring that none go unchecked. As the inspection progresses, AR Auto Detect visually shows the inspector each hole that has been inspected. The results of the inspection are then automatically serialized, identified and logged. This eliminates the need to manually enter data for each inspection. Not only can the metrics be logged, but photos or inspection equipment graphics can be saved.
This digitally captured information can be used electronically for individual assessment, and for all types of analysis that ensure optimal aircraft structural integrity. Information can be shared with key stakeholders in an organization and through its supply chain – at both detail and dashboard level – to improve outcomes for all parties involved.
Inspector Performance Assessment
AR Assessment is a powerful training and in-field capability that guides the inspector to assess errors during the evaluation process itself, by providing visual and audio suggestions based on what the inspector is doing. For example, during angle-beam inspection (also known as shear-wave inspection) which is used primarily for welds, the inspection probe is set at an angle. AI is used to interpret whether the inspector has set the probe at the correct angle. Suggestions can be made to the inspector as needed.
Assessment can also be used for defect accuracy. Each defect detected has an original signature – for example, severity, type and location. These defects can be grouped by category. During Assessment, AI can steer the technician to correctly identify the type of defect and make recommendations to the technician for validation. In essence, Assessment is an inspector’s “second set of eyes” while working on the job.
AR can take advantage of advanced features in wearable AR devices that use AI to track inspector’s actions, including both eye gazing and detailed hand movement. This data can also allow a supervisor to assess performance, give feedback, and speed the overall learning curve.
Best practices for implementing Augmented Reality in Non-Destructive testing include dual use mode that concurrently improves on-the-job performance and traditional training methods, along with well- designed AR that incorporates user-centric AI capabilities to support the inspector’s techniques while reducing effort and subjectivity. On the job, AR automatically captures data and provides information critical to maintaining aircraft health. AR helps NDT realize significant benefits including decreased training time, reduced time on task, standardized performance, and improved inspection results. Within the sphere of Industry 4.0 and Digital Transformation, AR is a powerful enabler of Industry 4.0 and Digital Transformation, providing benefits to technician and the many stakeholders throughout an organization’s supply chain.
In Augmented Reality Best Practices in Non-Destructive Testing Part 2, we will discuss the key factors for successfully moving AR into the enterprise, including data integration, equipment agnostic architecture, scalability considerations, how to best implement metaverse type collaboration, and standards.
About the Authors
Marlo Brooke is CEO & Founder of simpleAR®, software product provider of ARinspect™, and the simpleAR® enterprise AR authoring platform used widely by industry and government. Brooke is a widely published subject matter expert in augmented reality, digital transformation and supply chain.
Scott Toppel is President of AVATAR Partners®, a Platinum systems integrator of simpleAR® and provider of DoD and Industrial AR solutions. Toppel is a retired Navy CDR, and former combat and test pilot and maintenance officer and based in Virginia Beach, VA.