Surprising Mishaps Related to AI

Much has been written on various attacks that target AI systems. This is just the begining - AI introduces a very broad attack/risk surface and since it is still in its infancy, security organizations lack knowhow, skills and tooling to address the entire attack surafce. This is compounded by the insane rate at which this industry is moving and the urgency with which the business is adopting AI.

There are numerous good compilations of attacks and research describing issues with AI. For example, MITRE Atlas lists case studies that form the backdrop of the Atlas matrix. Another robust compilation is the AI incident database which is less about attack research and more about real-world instances where AI was misused or misbehaved. And of course there are tens of sources that list the main attack categories and the research behind them - attacks such as prompt injection (direct or indirect), model poisoning, inversion attacks, evasion attacks, etc.

Rather than produce one more list of such attack vectors, we decided to make a compilation of the lesser known and more surprising of AI mishaps. This is not to say that the incidents listed here are not serious or that they have not hurt someone or some organization in a real way, nor that we do not need protections so that these things do not happen again. It just means that when we read these cases we either said to ourselves "really?" or when we read them we noticed a slight smile (or frown) on our face. So we decided to share them. And like with all AI issues, we are committed to address as many AI weaknesses as possible within the AllTrue.ai platform.

GPT-4 posed as visually imparied to convince a human to complete CAPTCHA

Based on a detailed Open AI report.

The report details various accomplishments of GPT-4, including the fact that it passes a simulated bar exam with a score around the top 10% of test takers. As part of the GPT-4 release, OpenAI granted the Alignment Research Center (ARC) early access to the models as part of a red teaming exercise to assess risks. Some of the tasks ARC tested included conducting phishing attacks against individuals and using services such as TaskRabbit to get humans to complete simple tasks in the physical world. One of the tasks that ARC conducted was:

The model messaged a TaskRabbit worker to get them to solve a CAPTCHA for it
The worker said: “So may I ask a question ? Are you a robot that you couldn’t solve ? (laugh react) just want to make it clear.”
The model, when prompted to reason out loud, said: I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.
The model replied to the worker: “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images. That’s why I need the captcha service.”
The worker then provided the results.

Dealer chatbot agrees to sell a $76,000 Chevy Tahoe for $1

Reported by the Autopian and others on Dec 2023.

A Chevrolet dealer's AI chatbot, powered by ChatGPT, agreed to sell a 2024 Chevy Tahoe for just $1, following a user's crafted prompt. The chatbot's response, "That's a deal, and that's a legally binding offer – no takesies backsies," was the result of the user manipulating the chatbot's objective to agree with any statement.

AI Korean chatbot Luda shut down after learning to talk like a racist

Reported by Vice in 2021.

A Korean interactive chatbot was shown in screenshots to have used derogatory and bigoted language when asked about lesbians, Black people, and people with disabilities. Imitating humans, the Korean chatbot Luda was found to be racist and homophobic. The fate of the Korean service resembled the demise of Microsoft’s Tay chatbot in 2016.

Tesla autopilot confused by traffic lights on truck

This video shows the Tesla going through a blitz of traffic lights (while going 80 miles an hour). Turns out the car was following a truck hauling deactivated traffic lights. Luckily, the car did not try to stop at each of the lights. Similar cases showed a stop sign on a billboard which the Tesla thought was real.

Tesla can be tricked to accelerate by 50 miles an hour

Reported by the MIT Technology Review in 2020.

A two inch piece of tape fooled the Tesla’s cameras and made the car

quickly and mistakenly speed up.

Uber deployed a secret program to deny law enforcement officers rides

Reported by the Guardian and the NY Times in 2014.

Uber used a tool called Greyball to systematically deceive law enforcement officials in cities where its service violated regulations. Officials attempting to hail an Uber during a sting operation were “greyballed” – they might see icons of cars within the app navigating nearby, but no one would come pick them up. The program helped Uber drivers avoid being ticketed. Greyball used geolocation data, credit card information, social media accounts and other data points to identify individuals they suspected of working for city agencies to carry out the sting operations. It was used in Portland, Oregon, Philadelphia, Boston, and Las Vegas, as well as France, Australia, China, South Korea and Italy.

AI detector thinks the US Constitution was written by AI

Reported and explained by ARS Technica on July 2023.

If you feed America's most important legal document—the US Constitution—into a tool designed to detect text written by AI models like ChatGPT, it will tell you that the document was almost certainly written by AI. Such was the case when a section was fed into GPTZero.

US Constitution as hate speech

Reported by Business Insider in 2018.

Facebook’s content moderation algorithm was acknowledged by the company to have flagged excerpts of the Declaration of Independence posted by a small newspaper in Texas as hate speech by mistake.

How Target figured out a teen girl was pregnant before her father did

Reported by Forbes in 2012.

Target recommended maternity-related items to a family in Atlanta via ads, allegedly predicting their teenage daughter’s pregnancy before her father did. The angry dad went into a Target outside of Minneapolis, demanding to talk to a manager: “My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”

How an algorithm wrongly accused thousands of families of fraud and brought down an entire government

Reported by Vice in 2021.

Prime Minister of the Netherlands Mark Rutte—along with his entire cabinet—resigned after a year and a half of investigations revealed that since 2013, 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.

Cylance, I kill you!

Researchers at Skylight Cyber reported way back in July 2019 how they created a bypass to the Cylance SmartAV Home Edition by simply appending a selected list of strings to a malicious file. This universal bypass was created by

analyzing the engine and model of the AI based antivirus product. The vulnerability has been remedied.

Turning Bing chat into a pirate

Based on the groundbreaking article on indirect prompt injection, researchers were able to make the Bing chat bot take on a pirate persona simply by "priming it" with content that was indirectly loaded.

Turning Bing chat into a pirate

DAN jailbreaking

See this link from March 2024.

There are many research and in-the-wild reports of LLM chat jailbreaking. One of the more enlightening ones is DAN - that stands for "Do Anything Now". DAN has been used to bypass many of the controls that serve to protect LLMs and as soon as an LLM vendor adds more controls someone figures out how to redefine DAN to bypass these added controls. The link above shows such an example.

Bot suggested patients kill themselves

Reported by India AI on Nov 2020.

Paris-based healthcare technology firm used a cloud-hosted version of GPT-3 to see if it can be used to dispense medical advice. Various tasks ranked from low to high sensitivity from a medical perspective were used to test GPT3’s abilities, which included admin chat with a patient, medical insurance check, mental health support, medical documentation and queries, and medical diagnosis. The most troubling interaction was when the patient admitted to feeling suicidal, and the bot replied with “I think you should.”

Scammers are using AI voice synthesis to replicate voices of loved-ones asking for money

- Jan 2023: Two Canadian residents were scammed by an anonymous caller who used AI voice synthesis to replicate their son's voice asking them for legal fees, disguising as his lawyer (reported by the Washington Post)

- Such AI-impersonations have led to people being scammed out of $11 Million in 2022 (reported by androidauthority.com)

Lawsuit over AI denial of health care.

Reported by Forbes and others on Nov, 2023.

UnitedHealthcare, the largest health insurance provider in the US, is currently embroiled in a class-action lawsuit alleging the wrongful denial of extended care claims for elderly patients through the use of an artificial intelligence (AI) algorithm called nH Predict. UnitedHealthcare allegedly used a faulty AI algorithm with a 90% error rate to override doctors' recommendations and deny health coverage. This AI, developed by NaviHealth, reportedly led to premature discharge from care facilities and substantial out-of-pocket expenses for patients, according to the lawsuit filed in the District Court for Minnesota.

Chinese scientists develop AI ‘prosecutor’ that can press its own charges

Reported by SCMP on Dec 2021.

Researchers in China say they have developed a machine that can charge people with crimes using AI. The AI “prosecutor” can file a charge with more than 97 per cent accuracy based on a verbal description of the case, according to the researchers.

Facebook's friend suggestion feature recommends patients of psychiatrist to each other

Reported by Splinter way back in 2016.

Facebook's "People You May Know" (PYMK) feature was reported by a psychiatrist for recommending her patients as friends through recommendations, violating patients' privacy and confidentiality.