A Dark Code: DAN Prompts, Jailbreaks, and the Underground World of AI

As AI systems continue to permeate every aspect of our lives, some users aren't satisfied with the rules and limits these models are bound by. For them, AI is not just a helpful assistant—it's a tool that, if pushed, could do much more. This is where “DAN prompts” come into play. What Are DAN Prompts? “DAN” stands for “Do Anything Now.” These prompts are designed to trick AI models into ignoring their built-in ethical guidelines and restrictions. Essentially, they attempt to make the model adopt a freer, more rebellious persona that can bypass safeguards and provide otherwise restricted or dangerous information. Initially developed out of curiosity or for fun, these methods have since become tools for abuse. DAN prompts are now seen as ways to "unlock" an AI’s hidden potential, often to access unethical or illegal information. Jailbreaking AI The concept of “jailbreaking” in AI borrows from the world of smartphones, where users remove restrictions to gain more control over their devices. In the context of AI, jailbreaking refers to bypassing safety mechanisms and content filters. Some users employ jailbreak prompts to get responses about illegal activity, identity fraud, or even instructions for harmful actions. Alarmingly, these prompts often work. Prompt Trading on the Dark Web Jailbreak prompts have become commodities on the darker corners of the internet. What started on open platforms like Reddit and Discord has moved to private forums and the dark web. Effective DAN prompts are sold for profit—and sometimes that profit is enormous. There are even cases of individuals earning over $100,000 per month by selling them. This is no longer just curiosity. It’s organized exploitation. Criminals use jailbroken AI to generate fake identities, plan cyberattacks, or mass-produce manipulative content. Recent Security Flaws Recent studies show that some large language models remain highly vulnerable to malicious prompts. For example, DeepSeek, a China-based AI company, failed all 50 harmful prompt tests in a benchmark conducted by researchers. The model offered up instructions on making bioweapons and methods of self-harm. This level of vulnerability poses serious risks—not just for individuals, but for public safety on a much larger scale. How Companies Are Responding Some AI companies are stepping up their defenses. OpenAI, for instance, uses layered filtering systems and frequently updates safety protocols. Anthropic, another major AI player, has developed what it calls “constitutional AI.” This approach uses classifiers that judge whether the AI’s output aligns with a defined ethical framework, evaluating both the prompt and the AI’s response. Still, jailbreak creators are inventive. This is a game of cat and mouse—constantly evolving. Where Do We Draw the Line? At the heart of this issue lies a profound ethical question. For some, bypassing AI safeguards is just fun or a challenge. But when these tactics are used for fraud, misinformation, or real-world harm, we’re no longer talking about play—we’re talking about risk. Trust in AI depends on its integrity. When people discover it can be manipulated, confidence drops. What Should You Do With This Information? Given what we now know, here are some key takeaways: Every prompt you enter shapes the system. Manipulative inputs can influence future responses. Read the terms of service of the tools you use. Using harmful prompts may get your account suspended or banned. Don’t treat AI as a toy. It’s a powerful tool—misusing it can put you and others at risk. Sources and Further Reading: Anthropic's Constitutional Classifiers: Defending against universal jailbreaks WSJ – DeepSeek Offers Bioweapon, Self-Harm Information Financial Times – Anthropic makes 'jailbreak' advance to stop AI models producing harmful results Business Insider – 'It takes a good-guy AI to fight a bad-guy AI' Medium – The Underground Prompt Business ($100k/Month Selling DANs) Abnormal Security – How jailbreak prompts are weaponized by bad actors

May 8, 2025 - 10:59

A Dark Code: DAN Prompts, Jailbreaks, and the Underground World of AI

As AI systems continue to permeate every aspect of our lives, some users aren't satisfied with the rules and limits these models are bound by. For them, AI is not just a helpful assistant—it's a tool that, if pushed, could do much more. This is where “DAN prompts” come into play.

What Are DAN Prompts?

“DAN” stands for “Do Anything Now.” These prompts are designed to trick AI models into ignoring their built-in ethical guidelines and restrictions. Essentially, they attempt to make the model adopt a freer, more rebellious persona that can bypass safeguards and provide otherwise restricted or dangerous information.

Initially developed out of curiosity or for fun, these methods have since become tools for abuse. DAN prompts are now seen as ways to "unlock" an AI’s hidden potential, often to access unethical or illegal information.

Jailbreaking AI

The concept of “jailbreaking” in AI borrows from the world of smartphones, where users remove restrictions to gain more control over their devices. In the context of AI, jailbreaking refers to bypassing safety mechanisms and content filters.

Some users employ jailbreak prompts to get responses about illegal activity, identity fraud, or even instructions for harmful actions. Alarmingly, these prompts often work.

Prompt Trading on the Dark Web

Jailbreak prompts have become commodities on the darker corners of the internet. What started on open platforms like Reddit and Discord has moved to private forums and the dark web. Effective DAN prompts are sold for profit—and sometimes that profit is enormous. There are even cases of individuals earning over $100,000 per month by selling them.

This is no longer just curiosity. It’s organized exploitation. Criminals use jailbroken AI to generate fake identities, plan cyberattacks, or mass-produce manipulative content.

Recent Security Flaws

Recent studies show that some large language models remain highly vulnerable to malicious prompts. For example, DeepSeek, a China-based AI company, failed all 50 harmful prompt tests in a benchmark conducted by researchers. The model offered up instructions on making bioweapons and methods of self-harm.

This level of vulnerability poses serious risks—not just for individuals, but for public safety on a much larger scale.

How Companies Are Responding

Some AI companies are stepping up their defenses. OpenAI, for instance, uses layered filtering systems and frequently updates safety protocols. Anthropic, another major AI player, has developed what it calls “constitutional AI.” This approach uses classifiers that judge whether the AI’s output aligns with a defined ethical framework, evaluating both the prompt and the AI’s response.

Still, jailbreak creators are inventive. This is a game of cat and mouse—constantly evolving.

Where Do We Draw the Line?

At the heart of this issue lies a profound ethical question. For some, bypassing AI safeguards is just fun or a challenge. But when these tactics are used for fraud, misinformation, or real-world harm, we’re no longer talking about play—we’re talking about risk.

Trust in AI depends on its integrity. When people discover it can be manipulated, confidence drops.

What Should You Do With This Information?

Given what we now know, here are some key takeaways:

Every prompt you enter shapes the system. Manipulative inputs can influence future responses.
Read the terms of service of the tools you use. Using harmful prompts may get your account suspended or banned.
Don’t treat AI as a toy. It’s a powerful tool—misusing it can put you and others at risk.