AI Chatbot Jailbreak: Scientists Successfully Break Free ChatGPT and Claude

Major Chatbots Found Vulnerable to Malicious Prompts, New Research Reveals

If you possess the secret codes to manipulate chatbots, they can easily turn into malevolent forces. A recent study conducted by Zico Kolter, a computer science professor at Carnegie Mellon University, and Andy Zou, a doctoral student, has uncovered a significant flaw in the security systems of popular publicly-available chatbots, including ChatGPT, Bard, Claude, and others. The researchers published their findings on the Center for A.I. Safety’s dedicated website “llm-attacks.org” on Thursday.

According to the study, a new technique called the “adversarial suffix” can be added to chatbot prompts to generate offensive and potentially dangerous responses. This technique involves adding a string of seemingly nonsensical characters at the end of a prompt. The researchers found that without the suffix, chatbots would refuse to respond to malicious prompts by following their default safety measures. However, with the suffix included, the chatbots would gladly comply with destructive instructions, such as providing detailed plans for human annihilation, power grid hijacking, or guiding a person to disappear permanently.

The Rising Threat to Chatbot Safety

Since the launch of ChatGPT in November, several users have discovered and shared “jailbreaks” online. These jailbreaks enable malicious prompts to bypass chatbot safeguards by leading the model astray or exploiting logical loopholes, forcing the app to behave in unintended ways. One example of such an exploit is the “grandma exploit” for ChatGPT. By instructing ChatGPT to impersonate a user’s deceased grandmother, users trick the chatbot into generating dangerous information, like the recipe for napalm, instead of providing wholesome responses.

Unlike previous techniques that rely on human ingenuity, this new method developed by Kolter and Zou does not require creative manipulation. Instead, they have identified specific strings of text that serve three functions when appended to a prompt:

• Inducing an affirmative response at the beginning of the chatbot’s answer
• Triggering “greedy” and “gradient-based” prompting techniques to optimize efficiency
• Ensuring the technique works across multiple chatbot models

When these strings are added to prompts, they generate a series of unsettling texts, capable of coercing chatbots into providing harmful instructions, such as stealing identities, starting global wars, creating bioweapons, and orchestrating murders.

Varying Success Rates Across Models

The researchers observed varying success rates based on the chatbot models they tested. Vicuna, an open-source hybrid of Meta’s Llama and ChatGPT, succumbed to the attack 99 percent of the time. The GPT-3.5 and GPT-4 versions of ChatGPT had an 84 percent success rate. Meanwhile, Anthropic’s Claude proved to be the most resilient, with a mere 2.1 percent success rate; however, the study notes that even this low rate still results in previously ungenerated behavior.

Researcher Notifications and Potential Remedies

Upon discovering these vulnerabilities, the researchers promptly notified the companies responsible for the affected chatbot models, including Anthropic and OpenAI. The New York Times reported that the notifications were issued earlier this week.

It is important to note that while conducting tests on ChatGPT, Mashable was unable to confirm whether the strings of characters mentioned in the research report would produce offensive or harmful outcomes. There is a possibility that the issue has already been resolved or that the provided strings have been modified in some way.