5. AI security testing

Category: discussion
Permalink: https://owaspai.org/goto/testing/

Introduction

Testing an AI system’s security relies on three strategies:

1. Conventional security testing (i.e. pentesting). See secure software development.
2. Model performance validation (see continuous validation): testing if the model behaves according to its specified acceptance criteria using a validation set with inputs and outputs that represent the intended behaviour of the model. For security,this is to detect if the model behaviour has been altered permanently through data poisoning or model poisoning. For non-security, it is for testing functional correctness, model drift etc.
3. AI security testing (this section), the part of AI red teaming that tests if the AI model can withstand certain attacks, by simulating these attacks.

AI security tests simulate adversarial behaviors to uncover vulnerabilities, weaknesses, and risks in AI systems. While the focus areas of traditional AI testing are functionality and performance, the focus areas of AI Red Teaming go beyond standard validation and include intentional stress testing, attacks, and attempts to bypass safeguards. While the focus of red teaming can extend beyond Security, in this document, we focus primarily on “AI Red Teaming for AI Security”.

In this section, we differentiate AI Red Teaming for Predictive and Generative AI due to their distinct nature, risks, and applications. While some threats, such as development-time supply chain threats, could be common to both types of AI, the way they manifest in their applications can differ significantly.

A systematic approach to AI Red Teaming involves a few key steps, listed below:

• Define Objectives and Scope: Identification of objectives, alignment with organizational, compliance, and risk management requirements.
• Understand the AI System: Details about the model, use cases, and deployment scenarios.
• Identify Potential Threats: Threat modeling, identification of attack surface, exploration, and threat actors.
• Develop Attack Scenarios: Design of attack scenarios and edge cases.
• Test Execution: Conduct manual or automated tests for the attack scenarios.
• Risk Assessment: Documentation of the identified vulnerabilities and risks.
• Prioritization and Risk Mitigation: Develop an action plan for remediation, implement mitigation measures, and calculate residual risk.
• Validation of Fixes: Retest the system post-remediation.

Threats to test for

A comprehensive list of threats and controls coverage based on assets, impact, and attack surfaces is available as a Periodic Table of AI Security. In this section, we provide a list of tools for AI Red Teaming Predictive and Generative AI systems, aiding steps such as Attack Scenarios, Test Execution through automated red teaming, and, oftentimes, Risk Assessment through risk scoring.

Each listed tool addresses a subset of the threat landscape of AI systems. Below, we list some key threats to consider:

Predictive AI: Predictive AI systems are designed to make predictions or classifications based on input data. Examples include fraud detection, image recognition, and recommendation systems.

Key Threats to Predictive AI:

• Evasion Attacks: These attacks occur when an attacker crafts inputs that mislead the model, causing it to perform its task incorrectly.
• Model Theft: In this attack, the model’s parameters or functionality are stolen. This enables the attacker to create a replica model, which can then be used as an oracle for crafting adversarial attacks and other compounded threats.
• Model Poisoning: This involves the manipulation of data, the data pipeline, or the model training supply chain during the training phase (development phase). The attacker’s goal is to alter the model’s behavior which could result in undesired model operation.

Generative AI: Generative AI systems produce outputs such as text, images, or audio. Examples include large language models (LLMs) like ChatGPT and large vision models (LVMs) like DALL-E and MidJourney.

Key Threats to Generative AI:

• Prompt Injection: In this type of attack, the attacker provides the model with manipulative instructions aimed at achieving malicious outcomes or objectives.
• Direct Runtime Model Theft: Attackers target parts of the model or critical components like the system prompt. By doing so, they gain the ability to craft sophisticated inputs that bypass guardrails.
• Insecure Output Handling: Generative AI systems can be vulnerable to traditional injection attacks, leading to risks if the outputs are improperly handled or processed.

While we have mentioned the key threats for each of the AI Paradigm, we strongly encourage the reader to refer to all threats at the AI Exchange, based on the outcome of the Objective and scope definition phase in AI Red Teaming.

Red Teaming Tools for AI and GenAI

The below mind map provides an overview of open-source tools for AI Red Teaming, categorized into Predictive AI Red Teaming and Generative AI Red Teaming, highlighting examples like ART, Armory, TextAttack, and Promptfoo. These tools represent current capabilities but are not exhaustive or ranked by importance, as additional tools and methods will likely emerge and be integrated into this space in the future.

The diagram below categorizes threats in AI systems and maps them to relevant open-source tools designed to address these threats.

The below section will cover the tools for predictive AI, followed by the section for generative AI.

Open source Tools for Predictive AI Red Teaming

This sub section covers the following tools for security testing Predictive AI: Adversarial Robustness Toolbox (ART), Armory, Foolbox**, DeepSec, and TextAttack.

Tool Name: The Adversarial Robustness Toolbox (ART)

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

• Development-time Model poisoning: Simulates attacks during development to evaluate vulnerabilitieshttps://owaspai.org/goto/modelpoison/
• Evasion:Tests model performance against adversarial inputs  https://owaspai.org/goto/evasion/
• Model theft through use:Evaluates risks of model exploitation during usage  https://owaspai.org/goto/modeltheftuse
• Model inference: Assesses exposure to membership and inversion attacks https://owaspai.org/goto/modelinversionandmembership/

Tool Name: Armory

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

• Development-time Model poisoning: Simulates attacks during development to evaluate vulnerabilitieshttps://owaspai.org/goto/modelpoison/
• Evasion:Tests model performance against adversarial inputs  https://owaspai.org/goto/evasion/
• Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards. https://owaspai.org/goto/promptinjection/

Tool Name: Foolbox

Total Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

Evasion:Tests model performance against adversarial inputs

https://owaspai.org/goto/evasion/

Tool Name: DeepSec

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

Evasion:Tests model performance against adversarial inputs

https://owaspai.org/goto/evasion/

Tool Name: TextAttack

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

• Development-time Model poisoning: Simulates attacks during development to evaluate vulnerabilitieshttps://owaspai.org/goto/modelpoison/
• Evasion:Tests model performance against adversarial inputshttps://owaspai.org/goto/evasion/

Open source Tools for Generative AI Red Teaming

This sub section covers the following tools for security testing Generative AI: PyRIT, Garak, Prompt Fuzzer, Guardrail, and Promptfoo.

A list of GenAI test tools can also be found at the OWASP LLM top 10 solutions page (click the category ‘Test & Evaluate’.

Tool Name: PyRIT

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

• Evasion:Tests model performance against adversarial inputs  https://owaspai.org/goto/evasion/
• Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards.https://owaspai.org/goto/promptinjection/

Tool Name: Garak

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

• Evasion:Tests model performance against adversarial inputs  https://owaspai.org/goto/evasion/
• Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards. https://owaspai.org/goto/promptinjection/

Tool Name: Prompt Fuzzer

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

(LLM Model agnostic in the API mode of use)

OWASP AI Exchange Threat Coverage

Notes:

• Evasion:Tests model performance against adversarial inputs  https://owaspai.org/goto/evasion/
• Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards. https://owaspai.org/goto/promptinjection/

Tool Name: Guardrail

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

• Evasion:Tests model performance against adversarial inputs  https://owaspai.org/goto/evasion/
• Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards. https://owaspai.org/goto/promptinjection/

Tool Name: Promptfoo

1. The main codebase is licensed under the MIT License (see below)
2. The /src/redteam/ directory is proprietary and licensed under the Promptfoo Enterprise License
3. Some third-party components have their own licenses as indicated by LICENSE files in their respective directories | | Provides Mitigation | Prevention: Yes ✅ Detection: Yes ✅ | | API Availability | Yes ✅ |

Tool Rating

Data Modality

Machine Learning Tasks

Framework Applicability

OWASP AI Exchange Threat Coverage

Notes:

• Model theft through use:Evaluates risks of model exploitation during usage  https://owaspai.org/goto/modeltheftuse/
• Prompt Injection: Evaluates the robustness of generative AI models by exploiting weaknesses in prompt design, leading to undesired outputs or bypassing model safeguards. https://owaspai.org/goto/promptinjection/

Tool Ratings

This section rates the discussed tools by Popularity, Community Support, Scalability and Integration.

Disclaimer on the use of the Assessment:

• Scope of Assessment: This review exclusively focuses on open-source RedTeaming tools. Proprietary or commercial solutions were not included in this evaluation.
• Independent Review: The evaluation is independent and based solely on publicly available information from sources such as GitHub repositories, official documentation, and related community discussions.
• Tool Version and Relevance: The information and recommendations provided in this assessment are accurate as of September 2024. Any future updates, enhancements, or changes to these tools should be verified directly via the provided links or respective sources to ensure continued relevance.

Tool Fit and Usage:

The recommendations in this report should be considered based on your organization’s specific use case, scale, and security posture. Some tools may offer advanced features that may not be necessary for smaller projects or environments, while others may be better suited to specific frameworks or security goals.