Academy learning group 2 · Academy module (not an ISO clause) M03 · Chapter 3 of 8
Evaluate evasion and robustness carefully
About 5 minutes · ISO/IEC 27090:2026 · Published AI-security guidance
Session only. Enable saving or download a backup from this course’s hub.
Completion is self-reported and does not certify understanding.
See it · Try it · Recall it
Picture the topic
Remember this: A clean test set and an adversarial test ask different questions.
Explore the anchors
Define attacker access and permitted input changes.
Concept matching activity
Match this task or description to its concept:
Evaluate relevant manipulated inputs.
Choose a concept before checking the explanation.
A quick way to study this topic
- Hide the explanation and redraw these three anchors from memory.
- Explain the difference between the anchors in your own words.
- Apply the distinction to the worked example below. Check the full lesson for assumptions and exceptions.
- Recall it again tomorrow, then after a few days. A memory hint is a retrieval cue, not a replacement for understanding.
Original academy illustrations and fictional exercises. These are learning simulations, not a real virtual machine, network assessment or measure of job readiness. Activities reset when you leave or reload; they do not change quiz scores.
What this chapter explains
- Concepts and distinctions
- Apply the process and collect evidence
Concepts and distinctions
Evasion concerns inputs crafted at inference time to induce an unwanted result. The attacker’s capabilities matter: can they observe confidence scores, query repeatedly, alter a few input features or control an entire document? A robustness claim without these assumptions is difficult to interpret or reproduce.
Tests should distinguish ordinary distribution changes from intentional manipulation, while acknowledging that a single observation may not establish intent. A defense that helps against one tested input family can fail under other conditions. Accuracy on a clean benchmark, average performance across users and resistance to a defined adversarial test are different claims. None alone proves safety in every deployment context.
Apply the process and collect evidence
Use approved test datasets and controlled environments. State acceptable modifications, system versions and success criteria, then examine effects on legitimate users as well as adversarial cases. Consider rejection, escalation and human review when confidence is insufficient. Document what was not tested and assess whether the mitigation creates unacceptable false alarms or denial of service.
| Concept | What to remember |
|---|---|
| Threat model | Define attacker access and permitted input changes |
| Robustness test | Evaluate relevant manipulated inputs |
| Interpretation | Record coverage limits and operational consequences |
Worked example
A document classifier performs well on ordinary uploads but changes its decision when an attacker adds irrelevant-looking material. The proposed response is to reject all unusual documents automatically. Validate the threat assumption and affected behavior in a controlled test. Assess legitimate unusual documents and the consequences of rejection. Choose a response with suitable review and escalation, then retest both security effectiveness and ordinary service quality.
Check your understanding
What must accompany a claim that a model is robust against evasion?
Show explanation
The attacker assumptions tested modifications system version and evaluation limits. Robustness evidence is meaningful only within its defined conditions.
Source and study boundary
Official standard overview. The reference identifies the topic being taught. Explanations and examples are original academy material; a topic link does not establish independent approval or sufficient exam mastery.