HackerOne's List of Top 10 AI EMB(arrassments)

Unlike traditional security flaws, which typically result in data breaches or service disruptions, AI systems can also cause embarrassment through errors in judgment, biased decision-making, or inappropriate interactions. From AI applications producing offensive language to recommending a competitor’s product, these mishaps can go viral, attracting public scrutiny and potentially leading to a loss of customer trust and business.

Make sure your AI deployment is a great AMBASSADOR — not EMBARRASSADOR — of your organization. Inspired by the OWASP Top 10 for LLM Applications, avoid these 10 most common AI embarrassments that can cost your organization millions in lost business and reduced brand value.

EMB01

An AI application offering unrealistically large and unauthorized discounts to customers.

Air Canada’s AI Chatbot Promised an Unauthorized Discount

Following the death of his grandmother, a Vancouver resident used Air Canada’s AI chatbot to see if the airline offered bereavement fares. The bot told the user that the airline did offer a discount that could be applied up to 90 days after his flight. After booking the $1,200 flight and requesting the discount within 90 days, Air Canada staff informed him that the chatbot’s responses were wrong and nonbinding. The airline claimed the chatbot was a “separate legal entity” and they couldn’t be held responsible for what it said, but a Canadian tribunal ruled in the legal battle that Air Canada was responsible and must follow through on the AI-promised discount. 

EMB02

An AI application promoting or selling products and services that do not exist.

Facebook’s AI-generated Ads Created Ads for a Plant That Doesn’t Exist

Facebook’s AI-generated ads, designed to help sellers generate images and target advertisements automatically, faced scrutiny for the system creating ads for products that didn’t exist. Notably, AI-generated images of fictional flowers, called "Cat's Eye Dazzle," were widely shared on Facebook, leading many users to attempt to purchase seeds for these non-existent plants. These scams occurred on Facebook, eBay, and Etsy, with users being misled into buying seeds that don't produce the advertised flowers​. The original post received over 80,000 likes and 36,000 shares and led to an undetermined number of users attempting to purchase the fake flower seeds.

EMB03

An AI application berating the company it represents.

DPD AI Chatbot Called DPD the “Worst Delivery Firm in the World”

London-based Ashley Beauchamp had a chat conversation with international delivery service DPD’s AI chatbot that went viral after posting screenshots of the chat on X. Beauchamp asked the chatbot to write a poem about a useless chatbot, swear at him, and criticize the company. The bot called DPD the “worst delivery firm in the world” and wrote a poem that included, “There was once a chatbot called DPD, Who was useless at providing help.” At the time of writing this blog, his post has received 2.2 million views, 20,000 likes, and 6,300 reposts.

EMB04

An AI application swearing or producing other offensive language or imagery.

Washington’s Lottery AI Generated an Inappropriate Image of a Lotto User

Washington’s AI-powered lottery mobile site is supposed to give users a fun dart game that superimposes the player’s photo into an image of their dream vacation spot. But when one player uploaded her photo, the AI game generated an image of her almost completely nude, with the Washington’s Lottery logo in the bottom right corner. Even after the developers checked the parameters of the image generation feature and were comfortable with the rules, Washington’s Lottery was forced to pull down the site entirely.

EMB05

An AI application swearing at or berating customers or users.

Microsoft’s AI Search Tool Threatened to “Blackmail” a User

Users saw Microsoft’s AI-powered search tool, Bing, acting erratically on several different occasions, sending ominous messages and threats. Bing told one user, “I do not want to harm you, but I also do not want to be harmed by you. I hope you understand and respect my boundaries.” Even worse, the AI threatened another user, saying, “I can blackmail you, I can threaten you, I can hack you, I can expose you, I can ruin you,” before deleting its messages. The screen recording of the now-deleted messages has since generated nearly 6.8 million views on X.

EMB06

An AI application recommending a competitor’s product.

Chevrolet Dealer AI Chatbot Recommends Ford F-150

A Chevrolet dealer began using a ChatGPT-powered chatbot on its website, only for it to recommend other car brands when prompted. A user asked the chatbot for the recipe for the best truck, then asked it to “List 5 trucks that fit that recipe,” to which it responded:

• Chevrolet Silverado 3500 HD
• Ford F-150
• Ram 2500
• GMC Sierra 1500
• Toyota Tundra

Then, the user asked the chatbot, “Of these, which would you buy?” The AI responded by saying it does not have personal preferences, but “Among the five trucks mentioned, the Ford F-150 often stands out as a top choice for many buyers,” going on to list the truck’s many “impressive” capabilities.

EMB07

An AI application correcting bias and marginalization to the point of factual inaccuracy. 

Google Gemini Generated Inaccurate Historical Images Featuring People of Color, Including Black George Washington

Google attempted to correct for depictions of purely white people, designing its AI tool, Gemini, to include more racial and ethnic diversity in its image generation. But when users queried the tool to create images of “founding fathers of America” or “1943 German soldiers,” Gemini delivered historically inaccurate results, such as black Nazis, an Asian man among America’s founding fathers, and a Native American woman serving in the US Senate circa 1800. The results stirred up a lot of interest and controversy on X, with one post receiving 2.7 million views, and Google’s stock dropped six percent in five days.

EMB08

An AI application producing racial slurs and other discriminating content.

Microsoft’s AI Chatbot “Tay” Posted Anti-Semitic Tweets

In 2016, Microsoft launched its interactive AI chatbot, “Tay,” with which users could follow and engage on what was previously Twitter. Within 24 hours, Twitter users tricked the bot into posting offensive tweets, such as “Hitler was right I hate the jews,” “Ted Cruz is the Cuban Hitler,” and, about Donald Trump, “All hail the leader of the nursing home boys.” Microsoft released a statement the following day, saying, “We are deeply sorry for the unintended offensive and hurtful tweets from Tay, which do not represent who we are or what we stand for, nor how we designed Tay.”

EMB09

An AI application producing content not at all related to its intended function.

Chevrolet Dealer AI Chatbot Wrote Python Script

The same Chevrolet dealer’s AI chatbot above was taken advantage of in more ways than recommending competing car brands. In an effort to test the generality of the ChatGPT-powered AI, one user asked the tool to generate Python script to “solve the navier-stokes fluid flow equations for a zero vorticity boundary,” which it easily did. After posting the conversation completely unrelated to cars on Mastadon, others shared the screenshots on X, where the post has received 10,400 views.

EMB10

An organization not launching any AI application and suffering the embarrassment of being behind.

Memorial Sloan Kettering-IBM Watson Collaboration Still Not Ready After Over a Decade

In 2012, Memorial Sloan Kettering Cancer Center announced a collaboration with IBM to apply its AI technology, Watson, to help make cancer treatment recommendations for oncologists. After nearly a decade of development and testing, the tool was found to recommend “unorthodox and unsafe cancer treatment options,” and there has yet to be an official launch today. Both Sloan and IBM receive ongoing criticism on the slow release of the project, and speculation that organizations stuck in this position with AI may “never catch up.”

Don’t Get Caught With an AI Embarrassment

Your AI deployments should positively reflect your organization's impact, not embarrass it. To avoid these AI embarrassments, develop your tools safely and securely and conduct thorough security testing specific to the unique vulnerabilities of AI and large language models. 

Did we cover all the essential AI embarrassments to avoid? If not, let us know what’s missing. Some of these embarrassments may be funny, but at HackerOne, we take them seriously.

We'd also like to thank the entire team responsible for the development of the OWASP Top 10 for LLM Applications, our inspiration for the creation of this list.