Our hypothesis on AI Hallucinations is based on a quote from the OODAcon 2024 panel, “The Next Generative AI Surprise: “Artificial Intelligence hallucinations may sometimes provide an output that is a very creative interpretation of something or an edge case that proves useful.” With that framing in mind, the following is the first installment in our survey of differing perspectives on the threats and opportunities created by AI hallucinations. 

Background 

Following are working definitions of AI hallucinations generated by LLMs based on the prompt:  What is an LLM hallucination?

MattGPT

An LLM hallucination refers to instances where a language model generates information or responses that, while coherent and plausible-sounding, do not align with factual reality. This phenomenon is a byproduct of the model’s design to predict and generate text based on statistical patterns recognized in its vast training data. Essentially, these models excel at crafting sentences that mimic human language use, drawing on the immense corpus of text they’ve been trained on to produce responses that seem contextually appropriate. However, this capability does not equate to understanding the world or the veracity of the information being generated.

The term “hallucination” in this context is apt because it captures the essence of the problem: the model is creating something out of nothing, much like a mirage. It’s a fascinating yet challenging aspect of working with generative AI, as it requires users to critically assess the output of these models, understanding that while the language may sound convincing, it may be entirely fabricated or misleading.  

This characteristic of LLMs underscores a critical limitation in their current state of development:

• Despite their sophistication and the remarkable ability to generate human-like text, they lack the capacity for true comprehension or the ability to discern factual accuracy.
• This limitation is not merely a technical hurdle but a profound reminder of the distinction between the appearance of intelligence and genuine understanding.
• It raises important considerations for those of us relying on or developing these technologies, emphasizing the need for vigilance and the incorporation of mechanisms to verify the accuracy of generated content.

Understanding LLM hallucinations is crucial for anyone working with or relying on AI-generated content. It highlights the importance of maintaining a critical perspective on these models’ output. It reminds us that while AI can augment human capabilities in many ways, it also introduces new challenges that require careful navigation.

ChatGPT 4.0 

An “LLM hallucination” refers to instances where a large language model (LLM), like ChatGPT, generates plausible-sounding information that is incorrect or completely fabricated. This occurs due to how these models are trained and operated. Here are some key points about LLM hallucinations: 

1. Training on Patterns: LLMs are trained on vast amounts of text data and learn to predict the next word in a sequence based on patterns observed in the training data.  During operation, they cannot access external databases or the Internet or “understand” content as humans do.

2. Lack of Factuality: Since LLMs generate responses based on statistical likelihoods rather than factual verification, they can sometimes create responses that seem logical and coherent but are factually incorrect. This is particularly common when the model is asked about niche topics or specific details that may not be well-represented in the training data.

3. Context and Complexity: The accuracy of an LLM’s output can also depend on the complexity of the query and the context in which information is presented. Misinterpretations or slight errors in understanding the context can lead to responses that deviate from the truth.

4. Misinformation and Data Quality: The training data for LLMs can include inaccuracies, biases, and misinformation, which can be reflected in the model’s outputs. The model might also generate confident assertions about topics it has been trained with low-quality or incorrect data.

Due to these issues, while LLMs like ChatGPT are powerful tools for generating human-like text, their outputs must be critically evaluated, especially for tasks requiring high accuracy or reliance on up-to-date information. Users need to verify facts from reliable sources when accuracy is crucial. 

LLM Hallucinations:  Creative Interpretations of Something and Edge Cases that Prove Useful? 

According to the MIT Technology Review Emtech 2023, “In generative AI systems, “hallucinations” occur when large language models fabricate information, presenting fiction as fact. Research is underway at MIT to mitigate and eliminate the manufactured misinformation by generative AI systems in hopes of developing more trustworthy AI systems.” 

In the interim – before AI hallucinations are solved – if ever? – how are AI hallucinations possibly a creative interpretation of something or an edge case that proves helpful? The following is a broad spectrum of research findings on LLM hallucinations, with an eye towards a broad spectrum of “creative interpretations and edge cases” with benevolent and malevolent intent, as we are agnostic and open at this point in the generative AI onslaught. 

Can you trust ChatGPT’s package recommendations?

ChatGPT can offer coding solutions, but its tendency for hallucination presents attackers with an opportunity. Here’s what we learned.

In early 2023, VULCAN provided a creative edge case – for good or ill – for hacker access into developer ecosystems by way of LLM hallucinations:  

We’ve seen ChatGPT generate URLs, references, and even code libraries and functions that do not exist. These LLM (large language model) hallucinations have been reported before and may be the result of old training data.  If ChatGPT fabricates code libraries (packages), attackers could use these hallucinations to spread malicious packages without using familiar techniques like typosquatting or masquerading.   Those techniques are suspicious and already detectable. But if an attacker can create a package to replace the “fake” packages recommended by ChatGPT, they might be able to get a victim to download and use it.

The impact of this issue becomes clear when considering that. In contrast, previously, developers had been searching for coding solutions online (for example, on Stack Overflow); many have now turned to ChatGPT for answers, creating a major opportunity for attackers.

The attack technique – use ChatGPT to spread malicious packages

We have identified a new malicious package spreading technique we call “AI package hallucination.”  The technique relies on the fact that ChatGPT, and likely other generative AI platforms, sometimes answers questions with hallucinated sources, links, blogs, and statistics.

It will even generate questionable fixes to CVEs and – in this specific case – offer links to coding libraries that don’t exist.   Using this technique, an attacker starts by formulating a question asking ChatGPT for a package that will solve a coding problem. ChatGPT then responds with multiple packages, some of which may not exist. Things get dangerous when ChatGPT recommends packages not published in a legitimate repository (e.g., npmjs, Pypi, etc.). 

When the attacker finds a recommendation for an unpublished package, they can publish their malicious package in its place. The next time a user asks a similar question, they may receive a recommendation from ChatGPT to use the now-existing malicious package. We recreated this scenario in the proof of concept below using ChatGPT 3.5.

Industry Perspectives on the VULCAN Research Findings

Steven Zurier covered the VULCAN research findings and tracked down various industry respondents for some perspective: 

• Bar Lanyado, Security Researcher, Voyager18 Research Team at Vulcan Cyber:  “It’s very concerning how repetitive its answers are and how easily it responds with hallucinations,” said Lanyado. “We should expect to continue to see risks like this associated with generative AI and that similar attack techniques could be used in the wild. It’s just the beginning; generative AI tech is still pretty new. From a research perspective, we’ll likely see many new security findings in the coming months and years. That said, virtually all generative AI providers are working hard to decrease hallucinations and ensure that their products do not create cyber risks, and that’s reassuring.”

• Melissa Bischoping, Director of Endpoint Security Research at Tanium said companies should never download and execute code they don’t understand and haven’t tested—such as open-source GitHub repos or ChatGPT recommendations. Bischoping said teams should evaluate for security any code they intend to run and have private copies of it. “Do not import directly from public repositories such as those used in the example attack,” said Bischoping. “In this case, attackers are using ChatGPT as a delivery mechanism. However, compromising the supply chain through shared/imported third-party libraries is not novel.  “Use of this strategy will continue, and the best defense is to employ secure coding practices and thoroughly test and review code intended for use in production environments,” she continued. “Don’t blindly trust every library or package you find on the internet or in a chat with an AI.”

• Bud Broomhead, Chief Executive officer at Viakoo, added that this case serves as an example of yet another chapter in the arms race that exists between threat actors and defenders…“Ideally, security researchers and software publishers can also leverage generative AI to make software distribution more secure,” said Broomhead  The industry is in the early stages of generative AI being used for cyber offense and defense, Broomhead continued, who credited Vulcan and other organization with detecting new threats in time to prevent similar exploits.  “Remember, only a few months ago, I could ask ChatGPT to create a new piece of malware, and it would,” he said. It takes very specific and directed guidance to create it inadvertently—and hopefully, even that approach will be prevented by the AI engines soon.” 

For the full research analysis from the VULCAN team, go to this link. 

What Next?

How to spot AI Package Hallucinations

The VULCAN research suggests, “It can be difficult to tell if a package is malicious if the threat actor effectively obfuscates their work or uses additional techniques such as making a functional trojan package.  Given how these actors pull off supply chain attacks by deploying malicious libraries to known repositories, developers need to vet the libraries they use to ensure they are legitimate. This is even more important with suggestions from tools like ChatGPT, which may recommend packages that don’t exist or didn’t before a threat actor created them.  There are multiple ways to do it, including checking the creation date, number of downloads, comments (or a lack of comments and stars), and looking at any of the library’s attached notes. If anything looks suspicious, think twice before you install it.”

ChatGPT and Software Development

About the Author

Daniel Pereira

Daniel Pereira is research director at OODA. He is a foresight strategist, creative technologist, and an information communication technology (ICT) and digital media researcher with 20+ years of experience directing public/private partnerships and strategic innovation initiatives.