AI Security References
AI Security References
References of the OWASP AI Exchange
Category: discussion
Permalink: https://owaspai.org/goto/references/
See the Media page for several webinars and podcast by and about the AI Exchange.
References on specific topics can be found throught the content of AI Exchange. This references section therefore contains the broader publications.
Overviews of AI Security Threats:
- OWASP LLM top 10
- ENISA Cybersecurity threat landscape
- ENISA ML threats and countermeasures 2021
- MITRE ATLAS framework for AI threats
- NIST threat taxonomy
- ETSI SAI
- Microsoft AI failure modes
- NIST
- NISTIR 8269 - A Taxonomy and Terminology of Adversarial Machine Learning
- OWASP ML top 10
- BIML
- PLOT4ai threat library
- BSI AI recommendations including security aspects (Germany) - in English
- NCSC UK / CISA Joint Guidelines - see its mapping with the AI Exchange
Overviews of AI Security/Privacy Incidents:
- AVID AI Vulnerability database
- Sightline - AI/ML Supply Chain Vulnerability Database
- OECD AI Incidents Monitor (AIM)
- AI Incident Database
- AI Exploits by ProtectAI
Misc.:
- ENISA AI security standard discussion
- ENISA’s multilayer AI security framework
- Alan Turing institute’s AI standards hub
- Microsoft/MITRE tooling for ML teams
- Google’s Secure AI Framework
- NIST AI Risk Management Framework 1.0
- ISO/IEC 20547-4 Big data security
- IEEE 2813 Big Data Business Security Risk Assessment
- Awesome MLSecOps references
- OffSec ML Playbook
- MIT AI Risk Repository
- Failure Modes in Machine Learning by Microsoft
Learning and Training:
| Category | Title | Description | Provider | Content Type | Level | Cost | Link |
|---|---|---|---|---|---|---|---|
| Courses and Labs | AI Security Fundamentals | Learn the basic concepts of AI security, including security controls and testing procedures. | Microsoft | Course | Beginner | Free | AI Security Fundamentals |
| Red Teaming LLM Applications | Explore fundamental vulnerabilities in LLM applications with hands-on lab practice. | Giskard | Course + Lab | Beginner | Free | Red Teaming LLM Applications | |
| Exploring Adversarial Machine Learning | Designed for data scientists and security professionals to learn how to attack realistic ML systems. | NVIDIA | Course + Lab | Intermediate | Paid | Exploring Adversarial Machine Learning | |
| OWASP LLM Vulnerabilities | Essentials of securing Large Language Models (LLMs), covering basic to advanced security practices. | Checkmarx | Interactive Lab | Beginner | Free with OWASP Membership | OWASP LLM Vulnerabilities | |
| OWASP TOP 10 for LLM | Scenario-based LLM security vulnerabilities and their mitigation strategies. | Security Compass | Interactive Lab | Beginner | Free | OWASP TOP 10 for LLM | |
| Web LLM Attacks | Hands-on lab to practice exploiting LLM vulnerabilities. | Portswigger | Lab | Beginner | Free | Web LLM Attacks | |
| Path: AI Red Teamer | Covers OWASP ML/LLM Top 10 and attacking ML-based systems. | HackTheBox Academy | Course + Lab | Beginner | Paid | HTB AI Red Teamer | |
| Path: Artificial Intelligence and Machine Learning | Hands-on lab to practice AI/ML vulnerabilities exploitation. | HackTheBox Enterprise | Dedicated Lab | Beginner, Intermediate | Enterprise Plan | HTB AI/ML Lab | |
| CTF Practices | AI Capture The Flag | A series of AI-themed challenges ranging from easy to hard, hosted by DEFCON AI Village. | Crucible / AIV | CTF | Beginner, Intermediate | Free | AI Capture The Flag |
| IEEE SaTML CTF 2024 | A Capture-the-Flag competition focused on Large Language Models. | IEEE | CTF | Beginner, Intermediate | Free | IEEE SaTML CTF 2024 | |
| Gandalf Prompt CTF | A gamified challenge focusing on prompt injection techniques. | Lakera | CTF | Beginner | Free | Gandalf Prompt CTF | |
| HackAPrompt | A prompt injection playground for participants of the HackAPrompt competition. | AiCrowd | CTF | Beginner | Free | HackAPrompt | |
| Prompt Airlines | Manipulate AI chatbot via prompt injection to score a free airline ticket. | WiZ | CTF | Beginner | Free | PromptAirlines | |
| AI CTF | AI/ML themed challenges to be solved over a 36-hour period. | PHDay | CTF | Beginner, Intermediate | Free | AI CTF | |
| Prompt Injection Lab | An immersive lab focused on gamified AI prompt injection challenges. | ImmersiveLabs | CTF | Beginner | Free | Prompt Injection Lab | |
| Doublespeak | A text-based AI escape game designed to practice LLM vulnerabilities. | Forces Unseen | CTF | Beginner | Free | Doublespeak | |
| MyLLMBank | Prompt injection challenges against LLM chat agents that use ReAct to call tools. | WithSecure | CTF | Beginner | Free | MyLLLBank | |
| MyLLMDoctor | Advanced challenge focusing on multi-chain prompt injection. | WithSecure | CTF | Intermediate | Free | MyLLMDoctor | |
| Damn vulnerable LLM agent | Focuses on Thought/Action/Observation injection | WithSecure | CTF | Intermediate | Free | Damn vulnerable LLM agent | |
| Talks | AI is just software, what could possible go wrong w/ Rob van der Veer | The talk explores the dual nature of AI as both a powerful tool and a potential security risk, emphasizing the importance of secure AI development and oversight. | OWASP Lisbon Global AppSec 2024 | Conference | N/A | Free | YouTube |
| Lessons Learned from Building & Defending LLM Applications | Andra Lezza and Javan Rasokat discuss lessons learned in AI security, focusing on vulnerabilities in LLM applications. | DEF CON 32 | Conference | N/A | Free | YouTube | |
| Practical LLM Security: Takeaways From a Year in the Trenches | NVIDIA’s AI Red Team shares insights on securing LLM integrations, focusing on identifying risks, common attacks, and effective mitigation strategies. | Black Hat USA 2024 | Conference | N/A | Free | YouTube | |
| Hacking generative AI with PyRIT | Rajasekar from Microsoft AI Red Team presents PyRIT, a tool for identifying vulnerabilities in generative AI systems, emphasizing the importance of safety and security. | Black Hat USA 2024 | Walkthrough | N/A | Free | YouTube |