Machine‑learning (ML) models can behave predictably in testing and then unpredictably in the real world. In safety‑critical systems, unpredictability can present a serious safety risk. Mathematical analysis can help to reduce machine-learning (ML) safety risks.
Factors affecting machine-learning (ML) safety
Research shows that even well‑trained ML elements can behave unpredictably due to factors such as:
- Rare or adversarial inputs. Unexpected inputs can push models into extreme behaviours.
ref: Amodei et al., “Concrete Problems in AI Safety” https://arxiv.org/abs/1606.06565 - Embedded constraints. These can trigger timing‑related hazards.
ref: Sculley et al., “Hidden Technical Debt in Machine Learning Systems” https://research.google/pubs/pub43146/
- Distribution shift. Models fail silently when real‑world inputs drift away from the data they were trained on.
ref: Quionero‑Candela et al., “Dataset Shift in Machine Learning ” https://mitpress.mit.edu/9780262170055/dataset-shift-in-machine-learning/ - Sensor degradation and drift. Noise, corruption or ageing sensors distort inputs, causing models to make confident but unsafe predictions.
ref: Hendrycks & Dietterich, “Benchmarking Neural Network Robustness to Common Corruptions“ https://arxiv.org/abs/1903.12261
ref: Ovadia et al., “Can You Trust Your Model’s Uncertainty?” https://arxiv.org/abs/1906.02530 - Timing and compute limits. Under load or hardware constraints, models produce unstable outputs or miss critical deadlines.
ref: Breck et al., “The ML Test Score: A Rubric for ML Production Readiness” https://research.google/pubs/pub46555/ - Hybrid‑system interactions. ML outputs can violate assumptions in deterministic control logic, creating hidden unsafe behaviours.
ref: Leveson, “Engineering a Safer World” (STAMP/STPA)“ https://mitpress.mit.edu/9780262533690/engineering-a-safer-world/
These failure modes rarely appear in conventional testing but they can appear in deployed systems.
How mathematical analysis can help achieve machine-learning (ML) safety
Mathematical analysis can help to answer the safety questions arising from use of machine-learning (ML) elements:
| Risk Factor | Mathematical Analysis | What it Answers | References |
|---|---|---|---|
| Rare or adversarial inputs | Neural network verification / adversarial robustness verification | Can any bounded perturbation of the input cause an unsafe output? | Adversarial Robustness of Deep Neural Networks: A Survey from a Formal Verification Perspective [arxiv.org] https://arxiv.org/abs/2206.12227 |
| Rare or adversarial inputs | Reachability analysis of neural networks | What outputs can the network possibly produce for a range of inputs? | NNV’s “Learn Neural Network Verification in 3 Minutes” [zhongkuima.github.io] https://zhongkuima.github.io/guides/phase1/neural_network_verification/ |
| Distribution shift | Out-of-Distribution (OOD) detection | Is the system operating on data unlike anything seen during training? | “Out of Distribution Detection: Knowing When AI Doesn’t Know.” [sei.cmu.edu] https://sei.cmu.edu/blog/out-of-distribution-detection-knowing-when-ai-doesnt-know/ |
| Distribution shift | Statistical monitoring and uncertainty estimation | How can the system identify when confidence estimates are no longer reliable? | Survey of OOD detection and its role in safety assurance of autonomous systems. [arxiv.org], [arxiv.org] https://arxiv.org/html/2510.21254v1 |
| Sensor degradation and drift | Observer-based fault detection and isolation (FDI) | Can measured sensor values be distinguished from expected system behavior? | Introductory review of observer-based FDI, residual generation, and analytical redundancy. [zenodo.org] https://zenodo.org/records/22643351 |
| Sensor degradation and drift | Kalman filtering and state estimation | How much sensor error can be tolerated before state estimates become unreliable? | Aircraft FDI example explaining Kalman-filter-based sensor fault detection. [HW VS SW S…SERVER …] http://icas.org/icas_archive/ICAS2014/data/papers/2014_0455_paper.pdf |
| Embedded constraints | Worst-Case Execution Time (WCET) analysis | Will inference and control tasks always finish before their deadlines? | LDRA WCET overview for safety-critical embedded systems. [ldra.com] https://ldra.com/capabilities/wcet/ |
| Embedded constraints | Schedulability and worst-case performance analysis | Is the collection of tasks guaranteed to meet timing constraints under maximum load? | Cambridge overview of worst-case performance and time-predictable systems. [cambridge.org], [cambridge.org] https://www.cambridge.org/core/journals/research-directions-cyber-physical-systems/article/worstcase-performance-for-realtime-systems/844AC661DCC9F87148107C24411E6EC3 |
| Timing and compute limits | Delay-aware control stability analysis | Will execution delays, jitter, or dropped frames destabilize the control loop? | WCET and real-time systems literature. [link.springer.com], [cambridge.org] |
| Hybrid-system interactions | Hybrid-system verification | Can combinations of continuous dynamics and discrete decisions reach unsafe states? | Verification of Hybrid Systems surveys reachability analysis, model checking, abstraction, and logic-based verification. [georgejpappas.org], [symbolaris.com] https://www.georgejpappas.org/wp-content/uploads/2024/04/10.10072F978-3-319-10575-8_30.pdf |
| Hybrid-system interactions | Model checking | Does the controller satisfy formally specified safety properties for all behaviors? | Formal model checking overview, including cyber-physical and hybrid systems. [emergentmind.com] https://www.emergentmind.com/topics/formal-model-checking |
| Hybrid-system interactions | Reachability analysis | Can the closed-loop system ever enter a hazardous state? | CORA’s hybrid-system reachability tutorial. [tumcps.github.io] https://tumcps.github.io/CORA/pages/hybrid-systems/index.html |
| ML controller in the loop (cross-cutting) | NN-controlled-system reachability analysis | Can a neural-network-controlled plant reach an unsafe state over time? | ReachNN. [dl.acm.org] https://dl.acm.org/doi/fullHtml/10.1145/3358228 |
This analysis can help increase Machine-learning (ML) safety by providing:
- Insight into when ML behaves safely
- Clarity on when it becomes unpredictable
- Understanding of conditions that trigger unsafe behaviour
- Confidence integrating ML into high‑assurance systems
Click here to learn more about Matimar capabilities in mathematical analysis
or contact us to discuss how mathematical analysis can be applied to your particular system.