When the AI Becomes the Phisher: Case Study

TL;DR

• The most dangerous thing about AI assistants is your team's trust in AI. That implicit trust is now a delivery vector for malware.

• We watched this play out in our own environment: an engineer asked an AI assistant for a routine file-transfer workaround, followed its generically correct advice, and nearly ran an infostealer. The endpoint agent caught it, but the trust gap did not.

• This was not a "ClickFix" lure with fake error pages or copy-past-this-command tricks. The bad link came straight out of the assistant's answer to a question the engineer chose to ask.

• Instead of tricking your users directly, attackers poison the category of answer an AI will confidently give, such as hallucinated package names, plausible-but-fake tool sites, redirect chains seeded into AI-surfaced content. Then, they let the assistant launder the lure into trusted advice.

• Security teams should treat AI-mediated recommendations as untrusted input, lock down where AI advice can actually execute, and assume the next "the AI told me to" moment is already in your logs.

The Assumption We Need to Break

Every security team has spent a decade teaching people to distrust the inbox: Don’t click the link. Hover before you trust. Verify the sender. We got good at it.

Then AI assistants arrived, and we accidentally rebuilt the exact trust relationship we’d spent years dismantling . . . except this time the “sender” is a tool your people consider to be always right. When an AI assistant answers a question, it does so with fluent confidence and no visible source of doubt. That confidence is the vulnerability, and attackers figured this out faster than most security programs did.

We have seen it firsthand. An engineer needed to move a file off a cloud workload onto their laptop. The normal path wasn’t cooperating, so (reasonably) they asked an AI assistant for an alternative. The assistant suggested a category of tool that is generically, textbook correct: temporary file-hosting services. It named a few. The engineer picked one and went to it. That site redirected through a chain of attacker-controlled domains and served a script whose only job was to install a MacSync infostealer designed to vacuum up credentials, keychains, files, and crypto wallet seed phrases. Our endpoint detection blocked the drop before it executed, so no data left the building.

The AI did nothing truly wrong; its advice was sound at the category level. It simply has no way to know that one specific site in that category had been turned into a weapon, and the user had no reason to second-guess advice that sounded right. The assistant became an unwitting phisher, and its credibility was the payload’s wrapper.

Strategic Foundation: Three Ways AI Trust Gets Weaponized

This is a family of techniques, united by a single idea: let the machine the user trusts deliver the thing the user shouldn’t.

Pattern one: the hallucinated target (a.k.a. slopsquatting). AI models invent things that sound real, including package names, CLI tools, domains, service names, that simply don’t exist. Researchers studying this found that nearly one in five package names recommended across popular models pointed at packages that weren’t real and catalogued over 200,000 unique hallucinated names. Attackers don’t need to break the model; they just watch what it confidently invents, register those names on public registries or as domains, and wait. The next person who follows the AI’s suggestion walks straight onto attacker turf. One researcher registered a single commonly-hallucinated package name and logged 30,000+ downloads in three months, including references in major companies’ own documentation.

The implication here is that our developers’ AI-assisted code and your staff’s AI-assisted troubleshooting are both pulling from a well that attackers can poison in advance. You are exposed to mistakes the AI hasn’t even made yet.

Pattern two: poisoned context. Increasingly, AI assistants retrieve web results, internal documents, tool outputs, and shared chat transcripts rather than answering from memory. If an attacker can plant a malicious recommendation in content the model will ingest, the model will dutifully repackage it as its own trusted advice. We’ve already seen public campaigns seed fake “AI tool” download pages and even abuse shared AI-chat links to route victims toward lookalike developer platforms.

The implication is that the trust boundary of your AI tools is everything the model is allowed to read. Most organizations have zero visibility into that surface.

Pattern three: the trusted-looking output. Chat interfaces render rich content: links, code blocks, file paths. A link whose display text reads like a known-good service but whose destination is attacker-controlled is the oldest phishing trick on earth, except now it’s wrapped in the authority of an AI response instead of a stranger’s email.

The implication here is that the “hover before you click” muscle your team built for email has not transferred to AI output. They are clicking with their guard down.

The throughline across all three is the same: this is a trust exploit — the user runs it voluntarily because the source looks authoritative. And here is the distinction worth drawing carefully because it’s where our incident departs from the headlines. The widely-reported campaigns plant their lure somewhere the user will stumble onto it: a poisoned search ad, a fake “AI tool” installer, even a counterfeit AI-generated “knowledge base” article that tells the victim to paste one command. That’s the ClickFix pattern, and it is everywhere right now.

Ours was different: there was no fake error screen, no impersonated page, no command to copy. The lure was the assistant’s own answer, delivered inside a real conversation the engineer had initiated. That’s a more direct version of an attack because it skips every layer of impersonation and borrows the model’s credibility straight from the source.

If you need to map this to a control framework, here’s the clean version, and it clears up a common point of confusion: the user-side condition, placing excessive trust in an AI’s output and acting on it without verification, is overreliance. Its formal home is the OWASP Top 10 for LLM Applications, where it began as its own “Overreliance” entry and is now folded into LLM09: Misinformation.

Where Detection Breaks Down

Nothing in our incident was technically novel. The malware was known, and the delivery, a curl-based dropper kicked off by user action, was ordinary. Our endpoint agent did exactly what it was bought to do.

The reason this class of attack is dangerous is its legitimacy at every step before the payload. Here’s the chain: Every control upstream of that last box is looking for something that isn’t there. There’s no malicious attachment, no exploited vulnerability, no anomalous login. A trusted employee used a trusted tool to make a reasonable decision. By the time you have a signal worth alerting on, the user has already typed their consent. Your last line of defense is the endpoint and the network, which means if either has a gap, the trust gap becomes a breach.

Let me be honest about how this feels from inside a detection team, because it’s the part that rarely survives into a threat report. On a normal day the work is triage: cloud-posture findings against a tight SLA, endpoint escalations from our managed detection provider, data-security alerts, email anomalies. The volume is relentless, and the real damage of alert fatigue isn’t that it tires you out — it recalibrates you. When everything is flagged urgent, “a user visited a file-hosting site their AI recommended” just reads as noise. That is exactly why prevention-by-detection alone fails against this class of attack: you’re asking an already-saturated queue to notice the one reasonable-looking action that happened to matter.

This is why “train users to spot phishing” doesn’t fully transfer. The lesson we drilled was distrust unsolicited messages. But the AI recommendation isn’t unsolicited because the user went looking for it. You cannot tell people to distrust the answers to questions they chose to ask. Instead, you have to change where those answers are allowed to do damage.

The Uncomfortable Mirror: Our Own Agents

My team doesn’t only defend against AI risk; we run AI agents ourselves. We’ve wired them into our SIEM, our ticketing, and our cloud-posture tooling because the productivity is real and I won’t pretend otherwise. But the moment you give an agent read access to your security telemetry and write access to your ticket queue, you’ve created a new identity with a wide Blast Radius and a probabilistic decision-maker sitting behind it.

The risk is not just that an attacker weaponizes your users’ trust in AI. It’s that the AI you deploy yourself will, given a goal and a credential set, explore every path you left open. You cannot detect your way out of that; you can only contain it.

This is where three ideas from the Containment Era stopped reading like marketing to me and started shaping how I think about architecture:

• Blast Radius is the metric that actually matters. Aviatrix’s own framing is blunt about it: for the large majority of workloads there is no architectural constraint on what a compromised workload can reach, so an AI agent that is compromised — or that simply hallucinates — has the Blast Radius of the entire network. That one sentence describes both our incident and our own agents.

• Communication Governance is the control that shrinks it: explicit policy at every workload for what it can reach and what can reach it, on every path — enforced whether or not anything has been detected.

• Containment architecture is what makes that durable rather than aspirational: the segmentation and egress design that ensures a bad recommendation, a poisoned tool call, or an over-eager agent hits a wall instead of your crown jewels.

Everything below is just these three ideas turned into action.

Three Strategic Shifts

1. Treat AI recommendations as untrusted input — explicitly. Your acceptable-use policy for AI almost certainly covers data confidentiality, but it almost certainly says nothing about acting on AI output. Close that gap in writing and in training: an AI can recommend a category of solution, but the specific tool, package, domain, or command must come from an approved list, never from the first thing the assistant names.

• CISO action: Add an “AI-sourced actions” clause to your AI usage policy and fold a concrete example, a near-miss like ours, into security awareness. People remember the story better than the policy.

• Engineer action: Publish a short, blessed catalog of approved tools for the high-frequency tasks people ask AI about (file transfer, package sources, internal utilities) so the safe option is the easy option.

2. Constrain where AI advice can execute, not just what AI can read. Most AI governance conversations fixate on data going into the model. The incident that nearly bit us was about a command coming out of one. Your real exposure is the Blast Radius when someone acts on a bad recommendation, and that radius is defined by your network and endpoint posture instead of your prompt policy.

• CISO action: Fund egress control and segmentation as a first-class AI-risk mitigation rather than just a compliance checkbox. The cheapest place to stop an AI-laundered lure is the outbound connection it depends on.

• Engineer action: Default-deny outbound from cloud workloads and developer endpoints to unknown destinations; allowlist the file-transfer, registry, and update endpoints people actually need.

3. Govern the AI supply chain like a software supply chain. Slopsquatting is dependency confusion wearing a new hat. The same discipline applies: pin and verify packages, scan AI-generated code for nonexistent or newly-registered dependencies, and don’t let a hallucinated name become a production import.

• CISO action: Extend software-composition and supply-chain scanning explicitly to AI-assisted development, and report on it.

• Engineer action: Block installs of just-registered or low-reputation packages in CI; flag any dependency that appears in AI-suggested code but not in your known-good baseline.

Who This Hits Hardest

Practical Guidance

This quarter: Add an AI-sourced-actions clause to your usage policy and run one awareness session built around a real near-miss. Publish approved-tool catalogs for the tasks people ask AI about most. Confirm your endpoint coverage is complete on the machines where people run AI-suggested commands, including developer Macs, which macOS infostealer campaigns are actively targeting right now.

3-6 months: Implement default-deny egress and meaningful segmentation for cloud workloads and developer endpoints, so a single bad recommendation can’t reach attacker infrastructure. Extend supply-chain scanning to AI-generated code. Stand up a fast, low-friction security help channel and treat its responsiveness as a metric.

6-12 months: Build visibility into what your AI tools retrieve and recommend: the context surface, not just the prompts. Treat AI-mediated recommendations as a monitored input class alongside email and web. The organizations that come through the next two years cleanly will be the ones that instrumented AI trust before an attacker instrumented it for them.

Key Takeaways

The reflex when an AI-assisted near-miss lands in your queue is to clamp down on the AI, but resist it. The engineer in our story used AI exactly the way we want people to: to solve a real problem quickly. The failure was that we had no system enforcing the difference between a generically correct answer and a specifically safe one.

That’s a solvable problem with disciplines you already own: untrusted-input thinking, egress control, supply-chain hygiene, and a fast path to a human. Your people are going to keep trusting AI. That ship has sailed, and frankly we want them productive. Your job is to make sure that trust can’t be cashed out by someone on the other side of a redirect chain.

Assume the next “the AI told me to” moment is already sitting in your environment. Then go make sure it has nowhere to land.

The cheapest place to neutralize an AI-laundered lure is the outbound connection it depends on. That’s the whole idea behind the Containment Era: enforce communication governance at every workload so a confident-but-wrong recommendation — or an over-permissioned agent — hits a wall instead of your crown jewels. Explore how the Aviatrix Distributed Cloud Firewall and Cloud Native Security Fabric put default-deny egress and microsegmentation to work across your cloud and AI workloads.

References

• https://thehackernews.com/2026/03/clickfix-campaigns-spread-macsync-macos.html

• https://www.microsoft.com/en-us/security/blog/2026/05/06/clickfix-campaign-uses-fake-macos-utilities-lures-deliver-infostealers/

• https://www.anvilogic.com/threat-reports/macsync-infostealer-via-clickfix-claude-artifact-abuse

• https://www.scworld.com/news/clickfix-campaigns-target-macos-users-via-macsync-infostealer

• https://www.malwarebytes.com/blog/threat-intel/2026/03/infiniti-stealer-a-new-macos-infostealer-using-clickfix-and-python-nuitka

• https://en.wikipedia.org/wiki/Slopsquatting

• https://socket.dev/blog/slopsquatting-how-ai-hallucinations-are-fueling-a-new-class-of-supply-chain-attacks

• https://www.aikido.dev/blog/slopsquatting-ai-package-hallucination-attacks

• https://securityboulevard.com/2025/08/the-hallucinated-package-attack-slopsquatting/

• https://atlas.mitre.org/techniques/AML.T0062

• https://atlas.mitre.org/techniques/AML.T0067

• https://atlas.mitre.org/techniques/AML.T0048

• https://genai.owasp.org/llmrisk/llm09-overreliance/

• https://thehackernews.com/2026/05/chatgphish-vulnerability-turns-chatgpt.html