By Austin Amaya,
PhD, Director, Analytics & AI, ECS
and Matthew McDonald
Senior Director, D&I Technology and Innovation
The New Frontier of Risk: AI Models as Protected Assets
In 2025, the challenge isn’t building powerful AI models. It’s making sure they don’t become liabilities.
Across federal agencies, data scientists are training AI on some of the most sensitive information in existence: intelligence imagery, sensor feeds, and operational records. These models aren’t just smart systems; they’re restricted assets in their own right.
Why? Because unlike traditional software, AI doesn’t simply process data. It remembers it. And that memory can leak.
How AI Models Leak Sensitive Data
Even without access to raw training sets, adversaries can extract sensitive information directly from AI models themselves.
- Black-box attacks: By submitting repeated queries and studying outputs, attackers can infer whether specific data points were in training (membership inference) or even approximate sensitive images (model inversion).
- White-box attacks: With access to a model’s architecture and parameters, adversaries can analyze neurons and weights to reconstruct training data or reveal dataset properties such as hidden sensor characteristics.
The risk is clear: AI models trained on sensitive data inherit the same classification. Left unprotected, they can expose sources, methods, and capabilities, handing adversaries insights that years of intelligence collection sought to conceal.
ATTACK TYPE
Black-box attacks
(query-only)
WHAT HAPPENS
Adversary infers or reconstructs training examples
EXAMPLE RISK
“Was this satellite image in training?”
>> Leaks sensor characteristics
ATTACK TYPE
White-box attacks
(full access)
WHAT HAPPENS
Adversary mines neurons/weights for sensitive features
EXAMPLE RISK
“Which features drive this decision?”
>> Exposes vulnerabilities in critical infrastructure
From Cyber Defense to Model Defense
It follows that AI security requires the same rigor as cybersecurity. That means a layered defense posture, applied directly to models.
Differential Privacy:
Adds mathematical noise during training so no single example overly influences the model.
Data Scramblers (e.g., Disguised-Nets):
Transform inputs before training so models never “see” raw protected data.
Adversarial Testing:
Red-teaming with known attack methods to probe for leakage before deployment.
Output Sanitization:
Rounding or restricting outputs to make reverse-engineering harder.
Knowledge Distillation:
Training a “student” model from the outputs of a sensitive “teacher” model, reducing exposure of raw restricted details.
No one safeguard is enough. Agencies must combine techniques, just as they do with networks and infrastructure, to ensure resilience.
Can AI Models Trained on Sensitive Data Ever Be Downgraded?
Agencies often ask: if a model is trained on protected data, is it forever locked down?
The default answer must be yes. But in rare, mission-critical cases, downgrading may be considered, provided strict conditions are met.
- Define the sensitivity: Is classification tied to content (e.g., military platforms in imagery), source (e.g., sensor signatures), or metadata (e.g., resolution, geolocation)?
- Prove it won’t leak: Show that the model no longer contains or reveals the restricted element, backed by technical evidence.
- Validate with testing: Subject the model to adversarial red-teaming and formal privacy evaluation.
- Maintain oversight: Ensure ongoing monitoring and controls, just as with other sensitive systems.
Downgrading is possible, but it must be the exception, not the norm. Usability can never outweigh security.
The Future of Secure Federal AI
Getting this right requires expertise across multiple domains: machine learning, cybersecurity, and federal classification policy.
ECS helps agencies bridge these worlds by:
- Determining when models must remain sensitive, and when downgrading might be safely pursued
- Applying privacy-preserving training, adversarial testing, and secure deployment pipelines
- Conducting rigorous risk assessments and red-teaming
- Building trustable, usable AI that never compromises mission security
The next frontier of AI in national security is not just building smarter models, it’s building defendable models. Agencies that approach AI with the same discipline applied to cyber defense will unlock mission value while preventing mission risk.
