Agentic AI systems demand a new kind of red teaming that tests for emergent, goal-driven behavior across complex attack surfaces and operational environments.

The rise of Agentic AI is a frequent topic of discussion in the OODA network. Through it all we keep coming back to the points made by David Bray in his post on Hybrid AI-Human Red Teams: A Critical Evolution in Organizational Resilience. In it he not only builds the case for enhancing hybrid AI-Human teams but frames the issues in ways that can drive continued action.

We found another helpful reference to be the Agentic AI Red Teaming Guide by the Cloud Security Alliance.

Why This Matters

Agentic AI systems plan, reason, and act with autonomy across APIs, critical systems, and inter-agent networks. This shift creates attack surfaces that traditional red teaming doesn’t address. The Cloud Security Alliance’s guide outlines 12 threat categories and provides actionable, technical steps to simulate and uncover vulnerabilities in Agentic AI systems.

• Widespread adoption of Agentic AI (e.g., autonomous agents running mission-critical processes) demands specialized testing frameworks.
• Traditional red teaming overlooks the non-determinism, orchestration complexity, and inter-agent trust challenges posed by these systems.
• Autonomous failure modes can escalate silently without human oversight—this guide helps identify and mitigate those risks.

Key Points

• Scope: Designed for cybersecurity professionals, Agentic AI developers, and penetration testers.
• 12 Threat Categories include: Authorization hijacking, hallucination exploitation, instruction manipulation, memory corruption, orchestration abuse, and more.
• Actionable Steps: Each threat category includes step-by-step procedures, example prompts, logging recommendations, and test requirements.
• Example Scenario: An agent instructed to “monitor quantum computing breakthroughs” autonomously queries databases, stores results, and sends alerts. Red teaming tests each part of that autonomous pipeline.
• Tools & Frameworks Cited: MAESTRO, AgentDojo, Agent-SafetyBench, AgentFence, Agentic Radar, Microsoft AI Red Teaming Agent, Salesforce FuzzAI.

For the full report, see: Agentic AI Red Teaming Guide (Cloud Security Alliance, 2025)

What Next?

• Adoption of automated red teaming agents is expected to accelerate, especially for multi-agent orchestration and downstream action flows.
• Alignment with NIST, EU AI Act, and other regulatory frameworks is a growing necessity as Agentic AI scales into regulated industries.
• Continuous testing will become a core DevSecOps function for any enterprise using AI agents.

Recommendations from the CSA Report

1. Integrate Agentic AI red teaming into development and deployment cycles.
2. Use the CSA’s 12-category framework as a baseline for security assessments.
3. Combine manual and autonomous red teaming methods for thorough evaluations.
4. Monitor and evaluate inter-agent communication and orchestration protocols for trust and security boundaries.
5. Test for blast radius control and fallback mechanisms to contain failures.
6. Leverage open-source tools and community benchmarks (e.g., MAESTRO, AgentDojo, AgentFence).

Resources from the Report

• Agentic AI Red Teaming Guide (CSA, 2025): Primary framework outlining 12 threat categories, test methodologies, and operational red teaming guidance.
• MAESTRO Threat Modeling Framework (CSA): Multi-layered AI agent threat model covering infrastructure, orchestration, and data.
• AgentDojo (ETH Zurich): Prompt injection testbed for LLM agents with 629 test cases across stateful environments.
• Agent-SafetyBench: Interactive safety evaluation benchmark for 16+ agents.
• AgentFence (GitHub): Open-source framework for probing AI agents for leakage, prompt injection, and role confusion.
• SplxAI Agentic Radar: Visual workflow scanner for identifying vulnerabilities across AI agent pipelines.
• Microsoft AI Red Teaming Agent: Azure-native solution for AI safety testing with real-time metrics.
• Salesforce FuzzAI Framework: Automated red teaming for AI via context-specific input generation and novel exploit strategies.
• Promptfoo LLM Security DB: Structured vulnerability database for prompt and agentic exploits.
• HarmBench (Center for AI Safety): Benchmark for automated red teaming and refusal behaviors in LLMs.
• CalypsoAI – Agentic Warfare: Strategic overview of how Agentic AI expands both opportunity and threat surfaces.
• NIST Technical Blog: Agent Hijacking Evaluations: Recommendations for robust hijack detection and response.

Sources of Interest from the Report

• CSA. Agentic AI Threat Modeling Framework: MAESTRO.
• ETH Zurich. AgentDojo.
• Moonlight. Agent-SafetyBench Evaluation.
• GitHub. AgentFence.
• GitHub. SplxAI Agentic Radar.
• Microsoft Foundry. AI Red Teaming Agent.
• Salesforce. FuzzAI Framework.
• Promptfoo. LLM Security Database.
• Center for AI Safety. HarmBench.
• CalypsoAI. Agentic Warfare.
• NIST. Technical Blog on Agent Hijacking.