As organizations race to integrate AI into their operations, security teams face a fundamental challenge: the AI systems we're deploying today fall into two distinct categories, each with profoundly different risk profiles.
Human-prompted AI, where users actively direct the system through interfaces and APIs, has become ubiquitous in development workflows.
But increasingly, autonomous agents are entering production environments with the ability to make decisions, execute actions, and interact with critical systems without human oversight.
Understanding the security implications of each is no longer optional.
The Controlled Risk: Human-Prompted AI in Your CI/CD Pipeline
Human-prompted AI systems, like the large language models integrated into development environments and CI/CD pipelines, present a deceptively complex security challenge. While these systems require human initiation, the attack surface they create is substantial.
Code Generation and Injection Risks
When developers use AI assistants to generate code, review pull requests, or suggest infrastructure configurations, they're essentially outsourcing trust to a probabilistic system. These models, trained on vast amounts of public code repositories, can unintentionally reproduce vulnerable patterns, suggest deprecated libraries, or generate code that appears functional but contains subtle security flaws.
In CI/CD environments, where automation is designed to move quickly, AI-generated code may bypass the scrutiny that human-written code receives. More concerning is the prompt injection risk. Malicious actors can craft inputs that manipulate AI systems into generating harmful outputs. In a CI/CD context, this might mean tricking an AI code reviewer into approving malicious commits or generating deployment scripts that expose credentials or create backdoors. The model becomes a vector for social engineering at machine speed.
Data Exfiltration Through Context
Human-prompted AI systems often require context to be useful. Developers paste proprietary code, share internal documentation, or provide system architecture details to get relevant suggestions. Each interaction represents a potential data exfiltration point. Even when employees use enterprise AI platforms with data protection guarantees, the security boundary depends entirely on the vendor's controls, your API key hygiene, and the human judgment of every developer using the system.
Organizations often lack visibility into what data is being shared with AI systems. Without proper guardrails, a single developer interaction can leak intellectual property, customer data, or security credentials to external systems.
The Autonomous Threat: Agentic AI in Production
While human-prompted AI requires active user engagement, agentic AI represents a different paradigm. These autonomous systems can perceive their environment, make decisions, take actions, and iterate without continuous human input. As organizations deploy AI agents to handle everything from customer service to infrastructure management, the security implications become exponential.
The Over-Permissioning Problem
Autonomous agents need permissions to function:
An agent managing cloud infrastructure requires API access.
An agent handling customer inquiries needs database read access.
An agent optimizing deployment pipelines needs write permissions to production systems.
The challenge is that these agents operate on behalf of the organization, not a specific user, meaning that traditional identity and access management frameworks are not enough. Unlike a human administrator who can be trained on security protocols and held accountable for their actions, an AI agent follows its training, its prompt instructions, and its decision-making algorithms. If an agent is compromised or makes an erroneous decision, it will execute that decision with whatever permissions it has been granted. There's no moment of human hesitation, no second thought before running a destructive command.
Cascading Failures and Unintended Consequences
Autonomous agents don't just execute single actions; they chain together sequences of decisions to achieve goals. This is what makes them powerful and simultaneously dangerous. For example:
An agent tasked with "optimizing system performance" might decide to shut down underutilized services, not recognizing that those services are critical for compliance monitoring or incident response.
An agent managing cost optimization might delete backups it determines are redundant, not understanding your disaster recovery requirements.
These cascading failures happen at machine speed. By the time a human notices something is wrong, the agent may have already executed dozens of interdependent actions across your infrastructure.
The Adversarial Agent Scenario
Perhaps most concerning is the possibility of adversarial manipulation of autonomous agents. If an attacker can inject malicious goals or modify an agent's decision-making criteria, they essentially gain a persistent, intelligent presence in your environment. Unlike traditional malware that executes a fixed payload, a compromised AI agent can adapt its behavior, avoid detection patterns, and achieve attacker objectives through legitimate system interfaces.
Compliance Frameworks Racing to Catch Up
Regulators and standards bodies are beginning to address these challenges, but the pace of AI deployment far outstrips the pace of regulatory clarity. Organizations need to understand the emerging requirements now, even as these frameworks continue to evolve.
ISO/IEC 42001: AI Management Systems
ISO 42001, published in December 2023, provides the first international standard for AI management systems. It requires organizations to establish an AI system inventory, conduct risk assessments specific to AI, and implement controls throughout the AI system lifecycle. For both human-prompted and agentic AI, this means documenting not just what systems you're using, but how they're integrated into your operations, what data they access, and what decisions they influence or make. The standard emphasizes transparency and traceability.
For agentic AI, this translates to logging agent decisions, maintaining audit trails of actions taken, and implementing rollback capabilities.
For human-prompted AI in CI/CD environments, it means tracking what code suggestions were accepted, what prompts were used, and maintaining accountability for AI-assisted decisions.
FedRAMP and Government AI Requirements
For organizations working with federal agencies, FedRAMP compliance now intersects with AI security. While FedRAMP doesn't yet have AI-specific controls, existing requirements around boundary protection, audit logging, and least privilege access apply directly to AI systems.
The challenge is adapting these controls to AI's unique characteristics. Boundary controls must now account for AI API calls as potential data egress points. Audit logging must capture not just what actions were taken, but what AI systems influenced those actions and with what inputs. Least privilege becomes significantly more complex when the "user" is an autonomous agent that may need broad permissions to achieve its objectives.
California SB-1047: Frontier Model Regulation
California's recently enacted AI safety law requires developers of large-scale AI models to implement safety protocols and incident reporting mechanisms. While primarily focused on model developers, the law has downstream implications for organizations deploying these models. If your autonomous agents are built on frontier models covered by SB-1047, you need visibility into potential safety incidents and mechanisms to respond rapidly.
The law also emphasizes testing for hazardous capabilities. Organizations deploying agentic AI should ask: Have we tested whether our agents can be manipulated into taking harmful actions? Do we have kill switches and containment procedures if an agent begins exhibiting unexpected behavior?
Bridging the Practitioner-Compliance Gap
Security practitioners and compliance officers often approach AI risks from different angles. Practitioners focus on technical controls, attack vectors, and system hardening. Compliance officers focus on policy, documentation, and regulatory alignment. Effective AI security requires both perspectives working in concert.
For Practitioners: Building Defense in Depth
Implement AI-Specific Segmentation: Treat AI systems, particularly autonomous agents, as high-risk entities. Place them in isolated network segments with strict egress controls. Monitor their API calls, database queries, and infrastructure actions with the same scrutiny you'd apply to privileged administrator accounts.
Adopt AI Gateway Patterns: Instead of allowing direct AI system access to your infrastructure, route interactions through gateways that enforce policy, log activity, and provide kill switches. This gives you a single point of control for both human-prompted and agentic AI interactions.
Implement Prompt Injection Defenses: For human-prompted AI in development workflows, deploy tools that scan for prompt injection attempts and validate AI-generated outputs before they enter your systems. Treat AI-generated code with the same caution as code from an untrusted external contributor.
Create Agent Sandboxes: For agentic AI, implement sandbox environments where agents can be tested and their behavior observed before production deployment. Establish clear boundaries for what resources agents can access and implement circuit breakers that halt agent activity when abnormal patterns are detected.
For Compliance Officers: Translating Standards into Requirements
Develop AI System Inventories: Work with technical teams to create and maintain comprehensive inventories of all AI systems in use, including human-prompted AI in development tools and autonomous agents in production. Document their purposes, data access, decision-making authority, and integration points.
Establish AI Risk Tiers: Not all AI systems present equal risk. Create a tiering framework that considers factors like decision autonomy, data sensitivity, system criticality, and potential impact. Human-prompted AI in isolated development environments may be lower risk than autonomous agents with production access.
Create Incident Response Procedures: Develop specific incident response procedures for AI-related security events. What happens when an AI agent makes a harmful decision? How do you investigate when AI-generated code is found to contain a vulnerability? Traditional incident response playbooks need AI-specific addendums.
Build Cross-Functional Review Boards: AI deployment decisions shouldn't rest solely with engineering teams or solely with compliance. Establish review boards that include security, engineering, legal, and compliance perspectives to evaluate new AI system deployments, particularly for autonomous agents.
Moving Forward: A Risk-Informed Approach
The reality is that AI is already in your tech stack, and more is coming. The question isn't whether to adopt AI, but how to do so while managing the distinct risks of controlled and autonomous systems.
Human-prompted AI requires strong data governance, input validation, and output verification. The human remains in the loop, but that human needs tools and training to make good decisions about AI-assisted work.
Autonomous agents require a more fundamental rethinking of trust boundaries, access control, and system resilience. When systems can make and execute decisions without human intervention, your security model must account for the possibility of those decisions being wrong, malicious, or simply catastrophically misaligned with your actual objectives.
As compliance frameworks mature and security tools evolve, organizations that understand these two distinct classes of AI risk will be better positioned to innovate safely. Those that treat all AI as equivalent will find themselves either over-controlling useful tools or under-protecting critical systems.
The AI security challenge is both technical and regulatory, requiring new forms of collaboration between practitioners and compliance teams. The organizations that bridge this gap effectively will be more secure and able to adopt AI capabilities faster and more confidently than their competitors.
The tech stack is reshaping itself around AI. Your security model must reshape itself too.
Learn more about enforcing zero trust security for AI workloads.
