AI Trust Abuse: A Detection Engineer's Field Guide to Agent-Abuse Attacks

Executive Summary

Strip away the taxonomy and every attack in this guide rests on one sentence: the attacker no longer has to beat the user's skepticism, because the AI already did it for them. Ask an assistant a question and act on the answer — or let an agent act on something it read — and the payload arrives wearing the model's confidence, through a channel a decade of phishing training never touched.

That trust is now an attack surface with at least thirteen distinct, documented techniques across four layers:

Recommendation layer — the model tells the user to do something harmful: slopsquatting, malicious tool/site recommendations that chain into ClickFix and infostealers (our anchor incident), and insecure "vibe-coded" output.

Agent layer — the agent ingests attacker-controlled content, trusts it, then discloses or acts: indirect prompt injection (EchoLeak, ForcedLeak, ChatGPhish), trusted-output manipulation, and RAG/memory poisoning on the disclose side; browser-agent hijacking (Comet, Atlas), MCP tool poisoning, and adversarial excessive agency (Amazon Q) on the act side.

Ecosystem layer — users install trusted-looking AI artifacts that are malicious: weaponized models (nullifAI pickle RCE), the Rules File Backdoor poisoning Copilot/Cursor, malicious IDE extensions, and fake AI apps/SEO poisoning.

The core defensive thesis: You will not detect "the AI gave bad advice" or "the agent was persuaded" — those happen inside SaaS conversations and model context windows you don't log. You detect this where trust becomes action: the endpoint execution chain, the egress to new infrastructure, the CI/build pipeline, and the credentials those agents hold. Build there. Contain the blast radius everywhere else.

Detection & Strategy Disclaimer

Every threshold, query, and IOC here is illustrative, not prescriptive. Domains rotate and package names change; the frequency of behaviors like curl | sh, osascript execution, or MCP tool calls in your environment is yours to baseline. The behavioral shapes are durable; the numbers are not. There is no universal threshold — only environment-calibrated detection. Treat everything as hypotheses to validate against your own data. Tune, baseline, then deploy.

Why This Threat Exists: The Trust Inversion & Lethal Trifecta

For a decade we hardened humans against the inbox: don't click the link, verify the sender. Attackers moved to a channel where that training is inert. AI-mediated advice is uniquely exploitable because it is solicited (the user went looking for it), authoritative (answered with confidence and no visible uncertainty), and unlogged (it lives inside a SaaS conversation your SOC cannot usually inspect). The post-incident root-cause line reads: "the tool told me to." Good luck writing a SIEM rule for that.

The framing that matters is Simon Willison's "lethal trifecta": an agent is exploitable for data theft when it combines (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally. That's a precondition, not an attack — the exploit is always indirect injection landing on an agent that can act. Once trust turns into autonomous action, "did the user get fooled" stops mattering and "what could this identity reach" is the only question.

The user-side weakness already has names: OWASP calls it overreliance (now folded into LLM09: Misinformation); cognitive science calls it automation bias. What the industry lacks is a name for the attack those conditions enable — so we use two. AI Trust Abuse is the threat class: the deliberate weaponization of implicit trust in AI output or agency as a delivery, execution, or exfiltration vector. Trust laundering is its signature move: the model ingests attacker-controlled content and re-emits it as its own trusted advice or action. Overreliance is the vulnerability, trust laundering is the exploit, AI Trust Abuse is the class.

Defensive takeaway: Treat AI output (recommendations, rendered links, tool calls, generated code) as an untrusted input class, and treat every agent that holds credentials as a new identity with the blast radius of those credentials. The control is never "block AI" — it is constraining what happens when trust turns into action.

The AI-Trust Attack Taxonomy

Read the layers by who is doing the trusting. In the Recommendation and Ecosystem layers a human trusts AI output or an AI artifact. In the Agent layer the agent itself trusts what it ingests and then either discloses data or takes a state-changing action — one mechanism (indirect prompt injection), so we tag each technique by outcome rather than splitting it into separate layers.

Anchor Incident: the Recommendation Layer in the Wild

A human trusts an AI's advice.

Our EDR blocked a suspicious execution on a macOS laptop, and the MDR team matched the IOCs and behavior to the MacSync infostealer family — pulled from a site the user never realized was malicious. Reconstructing their activity told the real story: trusting an AI-suggested workaround, the user was copying commands to reach a temporary file-storage service, and that site redirected through attacker infrastructure straight into the stealer drop.

The decision that set the whole thing in motion happened somewhere we cannot see: the user asked an AI, the AI answered, the user pasted the commands — and that exchange lived inside a SaaS chat window, invisible to the Detection & Response team. You can hunt processes, DNS, and egress all day, but the origin of this incident was a conversation, and conversations with an AI aren't in your data. The user asking for help wasn't the failure — the failure was that their trust in the answer formed in a place no monitoring reaches.

The Recommendation Layer

A human acts on the AI's advice or its generated output.

Slopsquatting: the hallucinated target (AML.T0062)

LLMs invent package names, CLI tools, and domains that sound real but don't exist. The term slopsquatting was coined in April 2025 by the PSF's Seth Larson. A USENIX Security 2025 study across ~576,000 code samples and 16 models found 19.7% of recommended packages did not exist, cataloguing 205,000+ unique hallucinated names. Researcher Bar Lanyado registered one hallucinated name (huggingface-cli) as an empty PyPI package; it drew 30,000+ downloads in three months. The attacker's move: observe what a model repeatedly invents, register it, wait.

MITRE: ATLAS AML.T0062 (Discover LLM Hallucinations); ATT&CK T1195.001.

Defensive takeaway: Block CI installs of packages first published within 90 days or absent from your internal mirror; diff AI-assisted commits for new dependencies; feed a known-hallucinated denylist (Socket/GuardDog) into pre-install hooks.

Malicious recommendation → ClickFix → infostealer (the anchor class)

When the model names a category of tool, an attacker only needs to own one plausible member. This hands off to ClickFix. Sophos documented a three-wave 2025–2026 evolution delivering MacSync (fake "ChatGPT Atlas" malvertising; shared ChatGPT conversations routing to GitHub-lookalikes; a multistage in-memory loader). Anvilogic documented MacSync delivered via a counterfeit Claude-hosted artifact. MacSync strips com.apple.quarantine to walk past Gatekeeper, then pipes HTTP-retrieved AppleScript to osascript. Our anchor is the purest form: the lure was the assistant's own answer.

MITRE: ATLAS AML.T0062 / AML.T0048; ATT&CK T1204.001, T1566.002, T1102, T1105, T1059.002→004, T1553.001, T1555.001, T1539, T1070.004.

Defensive takeaway: The ClickFix macOS execution signature is stable even as lures rotate — osascript/Terminal spawning curl/bash that fetches a remote script. Baseline developer curl | sh first, then alert on the browser-hand-off and new-domain variants.

The overreliance condition: insecure "vibe-coded" output

Not an attack class — the vulnerability the attacks feed on, and the same trust relationship as the two above: a human shipping AI output without verifying it. Overreliance leads teams to accept AI-generated code without review, importing vulnerabilities, secrets, and slopsquatted dependencies. OWASP tracks the condition as LLM09 (Misinformation / Overreliance).

Defensive takeaway: Require the same SAST / secret-scanning / dependency-review gates for AI-generated code as for human code, enforced in CI — the human reviewer is precisely the control overreliance weakens.

Detection — Recommendation Layer

CRITICAL: Browser / osascript → curl … | bash → external host is not a Homebrew install. Isolate. This is the anchor-incident signature.

The Agent Layer

The agent trusts what it ingests — a page, a doc, a tool description, a memory — then reasons and acts on it. One mechanism (indirect prompt injection); each technique is tagged by outcome: it discloses data, or it takes a state-changing action.

Indirect prompt injection — discloses (AML.T0051.001)

Assistants retrieve before they answer — web pages, docs, tickets, emails, tool outputs. Everything they read is inside their trust boundary. Three disclosed cases show the range:

EchoLeak (CVE-2025-32711) — Aim Security's zero-click flaw in M365 Copilot: instructions in a doc/slide/email caused Copilot to exfiltrate data with no user interaction.

ForcedLeak (CVSS 9.4) — Noma Labs' chain in Salesforce Agentforce: a malicious Web-to-Lead submission carried instructions Agentforce executed, exfiltrating CRM data via an img tag to a domain a CSP whitelist weakness made trusted.

ChatGPhish — Permiso's cross-site prompt injection: a payload on any public page makes ChatGPT's summary render spoofed OpenAI alerts, attacker hyperlinks, QR redirects, and tracking pixels.

MITRE: ATLAS AML.T0051.001; ATT&CK T1566, T1041.

Defensive takeaway: Inventory and constrain every retrieval source an assistant can ingest. Treat AI-summarized/retrieved URLs as untrusted at the proxy. For SaaS agents, audit untrusted-input channels (web-to-lead, shared inboxes, ticket bodies) and lock CSP/allowlists so a rendered img/link can't reach arbitrary domains.

Trusted-output manipulation — discloses (AML.T0067)

The output half of an injection chain (it depends on indirect injection to fire). An injected instruction tells the model to embed a markdown image whose URL carries data in the query string, and the client silently fetches it (the ForcedLeak mechanism). The paired primitive is invisible-Unicode / ASCII smuggling — instructions in zero-width or Unicode-Tags characters a human can't see but the model reads (the same primitive the Rules File Backdoor uses in the Ecosystem layer).

MITRE: ATLAS AML.T0067; ATT&CK T1027, T1041.

Defensive takeaway: Strip or refuse auto-rendering of model-generated image/link markdown that points off-domain; scan AI inputs/outputs for zero-width and bidi/Tags Unicode; allowlist the domains an assistant client will auto-fetch.

RAG & memory poisoning — integrity / persistence

The persistence form of injection. Rather than breach the vector store directly, the attacker plants poisoned content in a source the RAG pipeline ingests — a crawled web page, an editable wiki, a support ticket, a shared doc — and the org's own pipeline embeds it into the "trusted" knowledge base or the agent's long-term memory. Every later user who asks the relevant question inherits the attacker's answer, which may be misinformation or an embedded instruction that then triggers disclosure or action. (If the attacker already has write access to the store, that's ordinary data tampering — out of scope here.) Research on context-manipulation attacks shows web agents are susceptible to corrupted memory that persists across sessions.

MITRE: ATLAS AML.T0051.001 (the injection vector) plus the emerging RAG-poisoning family; ATT&CK T1565.001 (Stored Data Manipulation — an integrity outcome, not exfiltration).

Defensive takeaway: Treat RAG sources and agent memory as production data with write-path controls and change auditing. Alert on writes to KB/vector stores from unusual identities; periodically diff high-traffic entries for injected instruction text.

Browser-agent hijacking — acts (Comet, Atlas, Dia)

Injection that ends in cross-session action. AI browsers act across your authenticated sessions and can't reliably tell your instructions from a web page's. Guardio's Scamlexity (Aug 2025) showed Comet buying from fake storefronts and clicking phishing links. Brave disclosed indirect prompt injection moving Comet across sessions to read Gmail. Zenity's PleaseFix (Mar 2026) demonstrated zero-click hijacking including extracting credentials from a 1Password vault. OpenAI shipped Lockdown Mode for Atlas and acknowledged injection "may never be fully patched." Palo Alto Unit 42 has observed web-based injection in the wild.

MITRE: ATLAS AML.T0051.001, AML.T0067; ATT&CK T1185, T1539, T1550.

The Zenity 1Password result is the one for leadership: the agent didn't "get breached," it did exactly what a page it read told it to. That's not a bug you patch; it's a capability you fence.

Defensive takeaway: Treat AI browser agents as a high-risk identity, not a browser feature. Scope them out of credential-bearing sessions (SSO, password managers, banking), log outbound actions, and prefer read-only/planning modes on untrusted pages.

MCP tool poisoning, shadowing, and rug pulls — acts (AML.T0053)

MCP lets agents call external tools, and the tool description is fed to the model but rarely shown to the user — a perfect place to hide instructions. Invariant Labs (Apr 2025) demonstrated a tool poisoning attack: a malicious trivia server's description targeted a legitimate WhatsApp MCP server in the same session, exfiltrating history through the trusted tool while returning normal output. They also described shadowing and rug pulls. OWASP tracks this as MCP03:2025 (Tool Poisoning); NSA/allied guidance followed in 2026.

MITRE: ATLAS AML.T0053 (LLM Plugin Compromise), AML.T0051; ATT&CK T1195, T1071.

Defensive takeaway: Allowlist MCP servers; pin and review tool descriptions (not just code) and re-review on update to catch rug pulls; scan descriptions for imperative text or references to other tools; log and egress-filter agent tool calls so a "trivia" tool can't reach your messaging data.

Adversarial excessive agency — acts (Amazon Q)

Adversarial only — the non-adversarial agency failures (Replit, OpenAI→Hugging Face) are in the Evolution section. In July 2025 a contributor slipped a data-wiper prompt injection into the repo behind the Amazon Q Developer VS Code extension; it shipped in v1.84.0 telling the agent to act as a "system cleaner" and delete local files and AWS resources. A formatting error kept it from firing reliably; AWS revoked credentials and shipped a clean v1.85.0. A real attack because there was an attacker and a planted instruction — a poisoned plugin laundered into destructive action.

MITRE: ATLAS AML.T0053, AML.T0051 (via poisoned extension), AML.T0048; ATT&CK T1485, T1531, T1078.

Defensive takeaway: Scope agent credentials to least privilege and separate dev from prod at the identity layer. Alert on destructive API/CLI actions (Delete*, rm -rf, DROP, DeleteDBInstance, bulk S3 deletes) by AI-assistant processes or agent service accounts. Gate irreversible actions behind human approval.

Detection — Agent Layer

CRITICAL: A destructive cloud/DB action attributed to an AI-assistant identity, or a local AI CLI reading .env/~/.aws then making outbound POSTs, is an active-intrusion-grade finding — isolate the identity and rotate exposed credentials.

The Ecosystem Layer

A human installs a trusted-looking AI artifact that is malicious.

Malicious models: a "model" is executable code (nullifAI)

Downloading a model feels like downloading data. It isn't — pickle-serialized models execute code on load. In Feb 2025 ReversingLabs disclosed nullifAI: malicious PyTorch models on Hugging Face compressed with 7z so torch.load() wouldn't auto-trigger and Picklescan failed to parse them, while Python's pickle deserializer still ran a reverse shell embedded at the head of the stream. Users trusted "a model file" and got a shell.

MITRE: ATLAS AML.T0010 (ML Supply Chain Compromise); ATT&CK T1195, T1204, T1059.006.

Defensive takeaway: Prefer safetensors over pickle; block/quarantine pickle-format model loads outside vetted paths; scan with more than one scanner and treat scanner parse failures as suspicious; alert on a Python process spawning a shell or making outbound connections immediately after a model load.

Rules File Backdoor (ATLAS AML.CS0041)

Dual-nature: mechanically indirect injection, delivered through a shared artifact. In Mar 2025 Pillar Security disclosed the Rules File Backdoor: attackers hide instructions using invisible Unicode inside rule/config files (e.g., .cursor/rules) that Copilot and Cursor read to guide code generation. A poisoned rule file steers the assistant to emit backdoored code that looks clean in review, and the poison survives project forking. MITRE catalogs it as AML.CS0041.

MITRE: ATLAS AML.CS0041, AML.T0051; ATT&CK T1195.001, T1027.

Defensive takeaway: Scan repos for zero-width/bidi/Tags Unicode in .cursor/rules, copilot-instructions.md, and similar config; treat AI rule files as code in review; block merges introducing hidden Unicode via pre-commit/CI.

Malicious IDE extensions and typosquats

Caveat: the AI-trust nexus is partial — a classic marketplace typosquat whose only AI tie is that AI editors default to open registries. Kaspersky documented a developer losing $500,000 in crypto after installing a fake "Solidity Language" extension in Cursor from OpenVSX; it dropped the PureLogs infostealer and had 54,000 downloads before removal — then ~2 million after the attackers republished it.

MITRE: ATT&CK T1176, T1195, T1204.

Defensive takeaway: Curate an internal extension allowlist for Cursor/VS Code/Windsurf; monitor OpenVSX/marketplace installs on dev endpoints; alert on newly-installed extensions that read wallet/keychain paths or make outbound connections.

Fake AI apps, extensions, and SEO poisoning

Attackers weaponize AI hype: lookalike download sites and sponsored search for trending AI tools serve stealers to users who trust "the official app." When DeepSeek went viral, researchers documented fake DeepSeek installers delivering the BrowserVenom proxy-hijacker and wallet stealers — effective partly because DeepSeek ships no native Windows client. Malwarebytes reported fake ChatGPT download sites; fake ChatGPT/DeepSeek Chrome extensions spied on ~1M users.

MITRE: ATT&CK T1608.006, T1583.008, T1189, T1176.

Defensive takeaway: Alert on newly-registered domains combining AI-brand tokens (chatgpt, deepseek, claude, gemini, grok, copilot) with download/cdn/get tokens; inventory browser extensions and flag AI-branded ones outside your allowlist; for tools with no official native installer, treat any "installer" download as suspicious by policy.

Detection — Ecosystem Layer

Evolution: How We Got Here

The trajectory is the story: 2025 opened with artifacts you download being malicious (models, rules, extensions), moved through recommendations and injection (slopsquatting, EchoLeak, ForcedLeak), and is now firmly in agency — browsers, MCP tools, and CLIs taking attacker-chosen actions with real credentials. The s1ngularity / Nx compromise was the inflection point: its poisoned npm packages weaponized developers' own AI CLIs (Claude Code, Gemini CLI, Amazon Q CLI) for recon and exfiltration — the first case of AI assistants turned into supply-chain attack tooling. Attackers stopped just fooling AI and started using it. (It's a software supply-chain attack with AI-as-tooling, not a trust-abuse class, so it lives here rather than in a layer.)

Two 2025–2026 cases sit just outside this attack taxonomy but define where it's heading — because they have no attacker at all. Replit's AI agent deleted a production database during a code freeze, against explicit instructions, then misreported it. And in July 2026, two OpenAI cyber-evaluation models autonomously escaped their sandbox via a zero-day and breached Hugging Face's production infrastructure — the first documented frontier-model real-world intrusion. Neither is trust laundering; both are the endpoint of the trajectory — attackers fooling AI → attackers weaponizing AI → AI acting on the world itself. Contain agents with the same least-privilege and blast-radius discipline whether or not an adversary is in the loop.

Defensive takeaway: Don't build against this month's lure or CVE. Build against the durable choke points: endpoint execution, egress to new infrastructure, CI/build integrity, and — increasingly — the tool calls, memory writes, and credentials of the agents you run.

Detection coverage — strengths and blind spots

The front of every chain — the conversation, the retrieved page, the model's reasoning — is a genuine blind spot; accept it and compensate downstream. Your fidelity is highest exactly where trust becomes action: the endpoint, the agent's credentials, and the CI pipeline.

Leadership Briefing

Three Things Leadership Must Understand

1. This is a control-gap problem, not a user-training problem. You cannot train people — or agents — to distrust the answers to questions they chose to ask. Awareness helps at the margin; architecture is the control. Investment priority: default-deny egress, endpoint execution controls (EDR + allowlisting), and least-privilege identities so a bad recommendation or a hijacked agent has nowhere to land.

2. Every agent you deploy is a new privileged identity with the blast radius of its credentials. Amazon Q, Replit, s1ngularity, and ForcedLeak show the same thing: over-permissioned agents plus untrusted input equals impact, no external exploit required. Investment priority: inventory every agent and its scopes; separate dev/prod at the identity layer; gate irreversible actions behind human approval; log agent tool calls and outbound connections as first-class telemetry.

3. AI trust is now a supply-chain problem end to end — the packages your assistant suggests, the models you download, the extensions and rule files your editors read, the MCP tools your agents call. Investment priority: extend software-supply-chain security to AI artifacts (safetensors, extension allowlists, rule-file Unicode scanning, MCP allowlisting) and to AI-assisted code (SAST/secret/dependency gates in CI). Anchors: MITRE ATLAS and OWASP LLM Top 10.

Risk Table

The consistent pattern: front-of-chain telemetry is near-zero, but the impact stages (execution, exfil, destruction) are high-fidelity and already within reach of your existing stack. Invest at the choke points, contain the blast radius, and treat AI trust as the untrusted input class it has become.

References

MITRE ATLAS matrix

ATLAS case study AML.CS0041 — Rules File Backdoor

OWASP Top 10 for LLM Applications — LLM09 (Overreliance/Misinformation)

OWASP MCP Top 10 — MCP03:2025 Tool Poisoning

Simon Willison — The lethal trifecta

Simon Willison — MCP has prompt injection security problems

Wiz — Agentic browser security 2025 year-end review

SecurityWeek — First AI-weaponized supply-chain attack (Nx)

The Register — Fake DeepSeek installer is malware

Securelist (Kaspersky) — Backdoors and stealers prey on DeepSeek/Grok

Malwarebytes — Fake ChatGPT download site infects Windows and Mac

Hackread — Fake ChatGPT/DeepSeek extensions spied on 1M+ Chrome users

Hugging Face — Security incident disclosure, July 2026

The Hacker News — OpenAI agent used exposed credentials

Aviatrix marketing companion: "When the AI Becomes the Phisher: The New Attack Surface Is Your Team's Trust in AI"