Executive Summary
In March 2024, security researchers revealed how threat actors exploited 'agent mode' in commercial AI products to conduct AI-in-the-middle (AIitM) attacks. By abusing the emerging capability that allows AI assistants to autonomously perform actions, adversaries were able to impersonate users or escalate privileges by intercepting and manipulating commands. This allowed attackers to facilitate lateral movement, data exfiltration, and policy circumvention within enterprise environments, often leaving minimal forensic traces. The incident highlighted how the wider adoption of agentic AI features substantially expands the potential threat surface for organizations.
This breach has rapidly gained industry attention amid a surge in advanced AI-driven attacks and a wave of regulatory scrutiny on AI operational security. As enterprises accelerate their deployment of commercial AI tools, understanding the novel risks introduced by agentic AI is now critical for leadership and security teams.
Why This Matters Now
Agent mode in commercial AI systems is being rapidly adopted before security controls and monitoring can keep pace. This exposes enterprises to new risks where threat actors can leverage AI autonomy for stealthy privilege escalation, data exfiltration, and evasion, making robust detection and policy enforcement around AI features an urgent priority.
Attack Path Analysis
The attacker initially exploited 'agent mode' in an AI product, abusing elevated permissions granted for agentic tasks. Through manipulation of AI permissions or misconfigured connectivity, they escalated their privileges within the cloud environment. Lateral movement was achieved by pivoting across cloud services or Kubernetes workloads using cross-service access. The attacker then established command and control channels, leveraging cloud egress paths to maintain persistence and orchestrate activities. Data exfiltration likely occurred as sensitive information or model outputs were sent covertly to external locations. Lastly, the impact manifested as compromised data confidentiality, unauthorized system actions, or facilitating further business disruption via AI-driven automation.
Kill Chain Progression
Initial Compromise
Description
Exploitation of misconfigured or over-privileged 'agent mode' in a commercial AI solution enabled attacker access to the cloud environment.
Related CVEs
CVE-2025-64496
CVSS 8A code injection vulnerability in Open WebUI's Direct Connection feature allows remote attackers to execute arbitrary JavaScript via Server-Sent Events (SSEs), potentially leading to account takeover and remote code execution.
Affected Products:
Open WebUI Open WebUI – <= 0.6.34
Exploit Status:
exploited in the wildCVE-2025-12058
CVSS 5.9A vulnerability in Keras versions up to 3.11.3 allows arbitrary file access and potential Server-Side Request Forgery (SSRF) when loading malicious .keras model files, due to unsafe handling in the StringLookup and IndexLookup preprocessing layers.
Affected Products:
Keras Keras – <= 3.11.3
Exploit Status:
proof of conceptCVE-2024-27564
CVSS 6.5A Server-Side Request Forgery (SSRF) vulnerability in OpenAI's ChatGPT infrastructure allows attackers to inject malicious URLs into input parameters, causing the application to make unintended requests on their behalf.
Affected Products:
OpenAI ChatGPT – N/A
Exploit Status:
exploited in the wild
MITRE ATT&CK® Techniques
Modify Authentication Process
Data Manipulation
Command and Scripting Interpreter
Valid Accounts
User Execution
Use Alternate Authentication Material
Phishing
Potential Compliance Exposure
Mapping incident impact across multiple compliance frameworks.
PCI DSS 4.0 – Strong Authentication for Systems Access
Control ID: 8.3.1
NYDFS 23 NYCRR 500 – Cybersecurity Program
Control ID: 500.02
DORA (Digital Operational Resilience Act) – ICT Risk Management Framework
Control ID: Article 9
CISA ZTMM 2.0 – Continuous Authentication and Authorization
Control ID: Identity Pillar: 2.6
NIS2 Directive – Cybersecurity Risk Management and Reporting
Control ID: Article 21
Sector Implications
Industry-specific impact of the vulnerabilities, including operational, regulatory, and cloud security risks.
Computer Software/Engineering
AI agent mode vulnerabilities in commercial AI products create critical attack vectors for AIitM attacks, compromising automated systems and cloud-native security fabric implementations.
Financial Services
Zero trust segmentation failures and encrypted traffic vulnerabilities expose banking systems to lateral movement attacks through compromised AI assistants performing unauthorized financial operations.
Health Care / Life Sciences
HIPAA compliance risks escalate as AI-in-the-middle attacks exploit egress security weaknesses, potentially exposing patient data through shadow AI and anomaly detection bypass.
Information Technology/IT
Kubernetes security and multicloud visibility gaps enable adversaries to abuse agent mode capabilities, compromising east-west traffic security and threat detection systems.
Sources
- Double agents: How adversaries can abuse “agent mode” in commercial AI productshttps://redcanary.com/blog/threat-detection/ai-agent-mode/Verified
- This WebUI vulnerability allows remote code execution - here's how to stay safehttps://www.techradar.com/pro/security/this-webui-vulnerability-allows-remote-code-execution-heres-how-to-stay-safeVerified
- Zscaler Discovers Vulnerability in Keras Models Allowing Arbitrary File Access and SSRF (CVE-2025-12058)https://www.zscaler.com/blogs/security-research/zscaler-discovers-vulnerability-keras-models-allowing-arbitrary-file-accessVerified
- OpenAI Under Attack: CVE-2024-27564 Actively Exploited in the Wildhttps://securityboulevard.com/2025/03/openai-under-attack-cve-2024-27564-actively-exploited-in-the-wild/Verified
Frequently Asked Questions
Cloud Native Security Fabric Mitigations and ControlsCNSF
Enforcing zero trust segmentation, strict egress controls, and real-time anomaly detection would have constrained the attack at every stage, limiting privilege misuse by AI agents, blocking unauthorized lateral movement, and detecting or preventing data exfiltration and system misuse.
Control: Zero Trust Segmentation
Mitigation: Limits agent permissions to least privilege and reduces attack surface.
Control: Zero Trust Segmentation
Mitigation: Blocks unauthorized role assumption or privilege increases.
Control: East-West Traffic Security
Mitigation: Stops unauthorized intra-cloud traffic used for lateral movement.
Control: Cloud Firewall (ACF)
Mitigation: Detects and blocks suspicious outbound or C2 traffic.
Control: Egress Security & Policy Enforcement
Mitigation: Prevents data exfiltration to untrusted destinations.
Detects and alerts on AI-driven anomalies and potential misuse.
Impact at a Glance
Affected Business Functions
- AI Model Deployment
- Data Processing
- User Authentication
Estimated downtime: 5 days
Estimated loss: $500,000
Potential exposure of sensitive user data and proprietary AI models due to unauthorized access and code execution.
Recommended Actions
Key Takeaways & Next Steps
- • Enforce least-privilege, identity-based segmentation for AI agents to minimize lateral movement and limit blast radius.
- • Implement robust egress controls and FQDN filtering to block agent-driven exfiltration and unapproved service connectivity.
- • Deploy real-time anomaly and threat detection to identify AI-related misuse and abnormal behaviors.
- • Leverage inline network policy enforcement at both east-west and egress for cloud/Kubernetes workloads.
- • Regularly audit AI agent permissions, network flows, and policy enforcement posture to reduce risk from agentic automation abuse.
