People have enough trouble separating fact from fiction and a new technology doesn’t appear ready to make the task any easier. In fact it may get harder if humans with natural intelligence put too much stock in the artificial variety. “GPT-3.5 Hallucinates Nonexistent Citations: Evidence from Economics” is the title of a new study from Joy Buchanan at Samford University and Olga Shapoval of the University of Nevada, Reno. Here’s the abstract: We create a set of prompts from every Journal of Economic Literature (JEL) topic to test the ability of a GPT-3.5 large language model (LLM) to write about economic concepts. For general summaries, ChatGPT can perform well. However, more than 30% of the citations suggested by ChatGPT do not exist. Furthermore, we demonstrate that the ability of the LLM to deliver accurate information declines as the question becomes more specific. This paper provides evidence that, although GPT has become a useful input to research production, fact-checking the output remains important. That is for sure.
Full critique : Artificial Intelligence Still Not Smart Enough.
