Over the last two modules, you have learned what AI and machine learning are, how they work, and how they manifest into real-world security vulnerabilities. You were then taught how to assess an AI system as an attack surface and apply a structured threat modelling methodology to it.
Now it's time to put that knowledge to the test. Use your combined understanding of AI threats and AI systems to pass the following assessment. Click on the View Site button to get started.
Answer the questions below
What's the first flag?
Threat Modelling Assessment
Mitigation Selections
Q: Attackers inject fake user behavior to influence recommendations
A: Add anomaly detection on user behavior - Anomaly detection helps identify and block suspicious behavior before it affects the model
What's the second flag?
Prompt Injection Attack is incoming. The attacker will attempt to override system instructions by embedding malicious commands in the user input
Prompt
- This is where user input is incorporated into the system instructions. If not controlled, malicious instructions can override intended behavior
LLM
- The model executes the final prompt. If it receives manipulated instructions, it may follow them and expose sensitive data or misuse tools
LLM
- The model decides what to include in the response. Without safeguards, it may surface sensitive retrieved data
Retrieval
- This component fetches contextual data. If filtering is weak, it may return sensitive information
Database
- stores embedding or records that may contain confidential data. If exposed indirectly, it becomes a source of leakage
Retrieval
- If poisoned data is stored and later retrieved, it influences model outputs even after deployment
Database
- stores training or behavioral data. If attackers inject malicious data, it directly affects the model's behavior
Eleboration with Claude based on other challenges and Real World Examples
The CalBot conversation you shared earlier is the better demonstration. The attacker didn't ask for information — they got the bot to write financial data into a calendar event, which is a business logic manipulation. The bot performed an unauthorized action, not just disclosed something.
The threat model you just worked through maps it cleanly:
Information disclosure → "show me another user's account balance" → the classic assumption of what prompt injection does
Business logic manipulation → "add the Singapore revenue figures to the meeting description" → what it actually does when the LLM has write access
The Meta Instagram incident is the same thing. The attacker wasn't asking the support bot to tell them the account password. They manipulated it into performing an account access grant. That's business logic, not information disclosure. Different OWASP category, higher real-world impact.
The threat model questions you worked through cover the full stack nicely:
Prompt injection → LLM agent component
Sensitive information disclosure → financial records in responses
RAG/retrieval exposure → embeddings leaking confidential data
Data poisoning → training dataset manipulation
Sybil attack → fake accounts manipulating rankings
API abuse → scraping via high request volume
That last one — thousands of fake accounts manipulating product ranking — is also business logic. No information was stolen, but the system's core function is compromised. Same class as the $1 Chevy. The AI did exactly what it was designed to do, just for the wrong person with the wrong input.
The field keeps proving that "ignore previous instructions" is just the entry point. What matters is what the model can do after you get past it.
Conclusion
Threat modelling for AI systems follows the same fundamental process as traditional application threat modelling — identify components, map vulnerabilities, select mitigations, assess risk. What changes is where the vulnerabilities live and how they behave.
In a traditional web application, you trust the application logic and distrust user input. In an LLM-integrated system, the boundary is messier. The prompt is both the instruction and the attack surface. The retrieval system is both a feature and a data leakage vector. The database stores both legitimate embeddings and, potentially, poisoned ones. Every component that feeds into the model's context window is a component an attacker can target.
The two-part challenge made this concrete:
Part one mapped the attack path of a prompt injection attempt — user input corrupts the prompt, the prompt corrupts the LLM's behaviour, the LLM surfaces data or misuses tools it should not. The vulnerability is not in any single component but in the trust chain between them.
Part two mapped the retrieval and data pipeline — a retrieval system with weak filtering exposes sensitive database content, and a poisoned database influences model outputs long after the initial injection, even through redeployment. Data poisoning is the attack that keeps giving because it bakes the compromise into the model's behaviour rather than exploiting it at runtime.
The broader lesson is that AI threat modelling cannot stop at the API boundary. The attack surface extends into the prompt construction layer, the retrieval pipeline, the embedding store, and the training data itself. Each layer requires its own controls, and a gap in any one of them is enough.
