Skip to main content

Algorithm Training and Real-World Environment Simulation

By Aaron Burciaga
Vice President of Artificial Intelligence
with Jason Lindauer
Modeling and Simulation Operations Manager
and Chad Fuller
Modeling and Simulation Technical Lead

Computer vision (CV) and algorithm training is a major challenge for the Department of Defense (DoD) as they look to make America a leader in military artificial intelligence (AI) applications. Fueled by the National Defense Authorization Act and executive orders, the DoD has increased funding research to develop this emerging technology. In order to train algorithms effectively, engineers need to create never-before-seen models and simulations (M&S) that accurately represent real-world environments (RWE). A lack of prior work, research, and real-world data often force engineers to take a “learn-as-you-go” approach to development, while COVID-19 has limited the ability to whiteboard ideas by turning all interactions virtual.

Working with DoD and partner organizations, ECS recently developed AI/CV algorithms to help autonomously control the flight and sensor capabilities of an unmanned aerial vehicle (UAV). Through the creation of a multidimensional, physics-based, simulation model, our experts have identified some best practices to ease the burden of AI/CV development.

Here are five key development practices to consider when implementing AI/CV algorithms:

Open System Architecture (OSA)

Incorporating OSA is a must for any AI/CV simulation that aims to model changing real-world physics. The architecture helps create a highly cohesive simulation environment that is flexible, scalable, and easily modified due to its modular nature. Without OSA, whenever environmental factors need to be changed, the simulation must be redesigned and rebuilt. But with the “plug-and-play” capabilities of OSA, modifying or reconfiguring a simulation is more efficient, resulting in a long-term reduction of operational costs.

Edge Computing and Mobility

As the needs of a mission evolve, AI/CV solutions need to be flexible, scalable, and mobile, but systems that require hardware buildouts can be cumbersome to expand. They often result in increased project costs and delayed scheduling timelines. Edge computing, however, enables engineers to better adapt the training and simulation environments to DoD’s operational needs. By bringing computation close to both data resources and the user, edge computing creates a mobile, flexible solution that offers:

Improved Resource Control
Faster Response Times
Decreased Bandwidth Usage
Lower Hardware Costs

With edge computing, organizations need not invest in costly hardware to run training and simulation environments. Instead, software is run on existing server networks or even in secure GovCloud. This flexibility enables simulations to run in different classification environments without the need to create a separate hardware buildout for each. Engineers can manage controlled unclassified information (CUI), secret, and top secret/sensitive compartmentalized information (TS/SCI) efficiently and securely all within the same network. In one case study, we found that utilizing a mobile environment resulted in a 20 percent reduction in the cost of procurement.

Computational Power: Central Processing Unit (CPU) vs. Graphics Processing Unit (GPU)

Running physics-based simulations is computationally intensive. While CPUs can handle a wide range of tasks quickly, they are limited in the number of tasks they can run concurrently. A GPU, however, is designed to quickly render high-resolution images and video at the same time.

In many ways, GPUs are faster and more efficient than traditional CPUs. They are better at collision detection, data preprocessing, and N-body calculations which predict the motion of individual bodies within a group of objects. For the physics-based simulations required by AI/CV models, GPUs are the better design choice for performing these critical functions.

Verification Validation & Accreditation (VV&A)

The goal of any simulation is to mimic or predict real-world behavior, but once a model is built, engineers need a way to test and measure its accuracy. VV&A is the method by which we evaluate those key metrics. Without effective VV&A, engineers cannot ensure a model accurately depicts a real-world environment, which nullifies any data the simulation uncovers. Many engineers falter in this crucial testing process.

According to DoD, VV&A is defined as:

VERIFICATION

The process of determining that a model and its data accurately represent the developer’s conceptual description and specifications.

VALIDATION

The process of determining the degree to which a model and data provide an accurate representation of the real world.

ACCREDITATION

The official certification that a model, simulation, and its associated data is acceptable for use for a specific purpose.

DoD requires that VV&A be used throughout the lifecycle of all its modeling and simulation (M&S) environments. Doing so ensures that any model built by DoD follows the Joint Artificial Intelligence Center’s (JAIC) standards for ethical AI.

VV&A has its challenges, especially with regards to validation, possibly the most important aspect of the VV&A triad. Since engineers are often evaluating new capabilities and technologies, they need to be able to determine if model accurately simulates a RWE even when no historical data exists. At ECS, we follow a framework put forth by Bob Eberth, former M&S advisor to the Marine Corps Warfighting Lab. In his words, “validation is the failure to invalidate.” If an engineer can uncover logic flaws that invalidate the model, the model is incomplete. The only way for an engineer to prove an AI/CV model is valid is through the effective testing and evaluation (T&E) of alternative hypotheses.

Platform and Sensor-Specific Requirements

Because simulations rely on synthetic data not obtained by direct, real-world measurements, it’s important to map and detail the requirements an algorithm needs to train effectively before building a simulation environment. Failing to lay out ontological and metadata standards for synthetic data can lead to delays and missteps down the line, especially in CV training. The earlier standards are established, the smoother the training process will be, and engineers will see a reduction in errors and misidentifications by the algorithm.

In the case of our UAV project, in order to properly create an effective simulation environment for training, we had to make sure that simulation met certain standards in target detection, tracking, and classification. For example, our synthetic data needed to take into account clutter effects on target images in order to properly train the detection algorithms against a litany of ground features. When building our simulation and training environments, we also needed to meet the design standards set by the Defense Advanced Research Projects Agency’s (DARPA) collaborative operations in denied environment (CODE). Using CODE as the foundation of our model environment enabled the UAV to better integrate and collaborate autonomously between other aerial systems.

While standardization may seem rudimentary, it often falls through the cracks and is not addressed until errors begin to occur. At that point, cost and schedule savings have been lost and retraining the algorithm is necessary.

A New Frontier of DoD Applications

Engineers have only begun to explore the possibilities of AI/CV within DoD, Homeland Security, and the government. The future of AI/CV will transform national security from supply chain management to autonomous vehicles and ships. As the technology matures and expands, ECS continues to lead its development. Using these “five key development practices,” ECS is creating new and advanced AI/CV solutions to power the next-generation of autonomous robotics and intelligent systems.

WE'RE HIRING