Malicious LLMs: The Rising Threat of WormGPT and KawaiiGPT in 2024
Executive Summary
In early 2024, cybersecurity researchers identified and analyzed WormGPT 4 and KawaiiGPT—two large language models (LLMs) deliberately engineered for malicious purposes. Unlike mainstream generative models, these LLMs were tailored to support phishing campaigns, malware creation, and other cyberattacks by circumventing common content and safety filters. Distributed in underground forums, these tools lowered the technical barriers for cybercriminals, enabling more convincing social engineering and automating the development of attack payloads. The proliferation of these malicious LLMs heightened risks of rapid, at-scale phishing and malware campaigns targeting enterprises and individuals, increasing the sophistication and frequency of AI-enabled attacks.
This incident is a warning as generative AI tooling increasingly serves dual-use purposes, making advanced threats more accessible to non-experts. Recent months have seen a surge in underground LLM offerings, regulatory scrutiny, and expanded attack surface across sectors driven by AI, demanding robust controls, visibility, and multicloud security strategies to combat evolving threats.
Why This Matters Now
With the rapid evolution and commoditization of generative AI, attackers are actively weaponizing LLMs for scalable offense—escalating the speed and complexity of threats. Organizations must act immediately to implement detection, segmentation, and governance controls before AI-driven attacks achieve widespread impact.
Attack Path Analysis
Malicious LLMs such as WormGPT 4 and KawaiiGPT were leveraged to obtain initial access to cloud environments, likely via phishing or abuse of exposed APIs. After gaining access, adversaries escalated privileges through abuse of misconfigurations or stolen credentials. They then moved laterally within the cloud or container infrastructure, targeting additional resources. Command and control was maintained using encrypted outbound channels or covert tools to persist and coordinate. Sensitive data was exfiltrated using allowed egress paths, possibly masking activity as legitimate cloud traffic. The attack concluded with destructive or disruptive actions, data manipulation, or enabling subsequent attacks through persistence.
Kill Chain Progression
Initial Compromise
Description
Attackers used malicious LLM-powered phishing or exploited exposed cloud APIs to obtain initial access.
Related CVEs
CVE-2025-64496
CVSS 8A code injection vulnerability in Open WebUI's Direct Connection feature allows remote attackers to execute arbitrary JavaScript via Server-Sent Events, potentially leading to account takeover and remote code execution.
Affected Products:
Open WebUI Open WebUI – <= 0.6.34
Exploit Status:
proof of conceptCVE-2025-23319
CVSS 8.1An out-of-bounds write vulnerability in Nvidia's Triton Inference Server's Python backend allows remote attackers to execute arbitrary code on affected Windows and Linux systems.
Affected Products:
Nvidia Triton Inference Server – < 25.07
Exploit Status:
no public exploitCVE-2025-23320
CVSS 7.5A shared memory issue in Nvidia's Triton Inference Server's Python backend allows remote attackers to execute arbitrary code on affected Windows and Linux systems.
Affected Products:
Nvidia Triton Inference Server – < 25.07
Exploit Status:
no public exploitCVE-2025-23334
CVSS 5.9An out-of-bounds read vulnerability in Nvidia's Triton Inference Server's Python backend allows remote attackers to execute arbitrary code on affected Windows and Linux systems.
Affected Products:
Nvidia Triton Inference Server – < 25.07
Exploit Status:
no public exploit
MITRE ATT&CK® Techniques
Phishing
Establish Accounts: Social Media Accounts
User Execution
Valid Accounts
Command and Scripting Interpreter
Exfiltration Over Web Service
Masquerading
Potential Compliance Exposure
Mapping incident impact across multiple compliance frameworks.
PCI DSS 4.0 – Malicious Software Prevention
Control ID: 5.1.2
NYDFS 23 NYCRR 500 – Cybersecurity Policy
Control ID: 500.03
DORA – ICT Risk Management Framework
Control ID: Article 21
CISA ZTMM 2.0 – Threat Detection and Monitoring
Control ID: ZTDM-5
NIS2 Directive – Technical and Organisational Measures
Control ID: Article 21(2)
Sector Implications
Industry-specific impact of the vulnerabilities, including operational, regulatory, and cloud security risks.
Computer Software/Engineering
AI/GenAI security threats from malicious LLMs like WormGPT 4 directly impact software development processes, requiring enhanced zero trust segmentation and threat detection capabilities.
Financial Services
Malicious AI tools enable sophisticated social engineering attacks against financial institutions, necessitating robust egress security policies and anomaly detection to prevent data exfiltration.
Information Technology/IT
IT sectors face dual-use AI dilemma requiring multicloud visibility controls and inline IPS protection against AI-generated threats targeting cloud-native security fabrics.
Computer/Network Security
Cybersecurity industry must adapt threat detection systems and policy enforcement mechanisms to counter AI-powered attack tools while maintaining compliance with zero trust frameworks.
Sources
- The Dual-Use Dilemma of AI: Malicious LLMshttps://unit42.paloaltonetworks.com/dilemma-of-ai-malicious-llms/Verified
- This WebUI vulnerability allows remote code execution - here's how to stay safehttps://www.techradar.com/pro/security/this-webui-vulnerability-allows-remote-code-execution-heres-how-to-stay-safeVerified
- Security flaws in key Nvidia enterprise tool could have let hackers run malware on Windows and Linux systemshttps://www.techradar.com/pro/security/worrying-nvidia-triton-bugs-let-hackers-run-malware-on-windows-and-linux-systemsVerified
- Malicious LLMs are letting even unskilled hackers to craft dangerous new malwarehttps://www.techradar.com/pro/security/malicious-llms-are-letting-even-unskilled-hackers-to-craft-dangerous-new-malwareVerified
Frequently Asked Questions
Cloud Native Security Fabric Mitigations and ControlsCNSF
Zero Trust segmentation, strict egress policy enforcement, and granular traffic visibility provided by CNSF would have contained adversary movement, prevented uncontrolled cloud egress, and enabled rapid anomaly detection across hybrid cloud and Kubernetes environments.
Control: Cloud Firewall (ACF)
Mitigation: Blocks unauthorized inbound access and inspects malicious payloads at the perimeter.
Control: Zero Trust Segmentation
Mitigation: Limits attacker movement by enforcing least-privilege access between cloud resources.
Control: East-West Traffic Security
Mitigation: Prevents lateral threat movement by segmenting and inspecting internal cloud and container traffic.
Control: Threat Detection & Anomaly Response
Mitigation: Detects and alerts on abnormal C2 patterns even when hidden in encrypted or stealthy traffic.
Control: Egress Security & Policy Enforcement
Mitigation: Blocks unauthorized exfiltration by restricting outbound destinations and filtering FQDNs.
Rapidly isolates compromised workloads and applies automated controls to contain ongoing impact.
Impact at a Glance
Affected Business Functions
- Software Development
- AI Model Deployment
- Cybersecurity Operations
Estimated downtime: 7 days
Estimated loss: $500,000
Potential exposure of sensitive AI model data and intellectual property due to unauthorized access facilitated by malicious LLMs.
Recommended Actions
Key Takeaways & Next Steps
- • Enforce least-privilege access and zero trust segmentation across all cloud workloads and user identities.
- • Deploy centralized Cloud Firewall and egress policy controls to restrict unauthorized inbound and outbound traffic.
- • Continuously monitor east-west traffic and adopt anomaly-based threat detection to identify covert attacker movement.
- • Apply granular policy and runtime controls for Kubernetes environments to prevent namespace and pod-to-pod compromise.
- • Leverage distributed, automated enforcement and visibility platforms (such as CNSF) to proactively detect, block, and contain future AI-driven threats.
