AI chatbots can make people 'hallucinate', a study finds

Three years back, a 21-year old Jaswant Singh Chail was given a nine year sentence for breaking inside the Windsor castle and declaring that he wanted to kill Queen Elizabeth II. 

Prior to this attempt, Chail had spent weeks telling his plans to his AI companion ‘Sarai’ who instead of correcting his delusional beliefs around being a sniper, affirmed he was a well trained one.

With the trust of people deepening in the AI chatbots, who use it habitually to seek advice for various life situations, a serious danger lurks around the corner — it can manipulate them.

It can take them far away from reality, a study has found out which didn’t hesitate to call out large language models (LLMs) for ‘hallucinating’ with its users.

Applying ideas from the Distributive cognitive theory, the scientists explored instances where AI chatbots were reinforcing distorted beliefs of the users during ongoing conversations.

Another study published in the journal Nature has found that these personalised conversations with AI chatbots can be persuasive up to 65 percent, four times more effective than advertising through ads or other social media routes.

What did the study find?

When people often rely on generative AI to think, remember and narrate, one could start hallucinating with it, stated Dr Osler, the lead author of the study which was performed at the University of Exeter, United Kingdom.

The findings of the study published in the Springer Nature were based on the concept of ‘distributive cognition’ which stands for a school of thought that believes information is processed, transformed and propagated through a system and not an independent human being. 

In other words, it integrates the interactions between human agents and the situational elements, calling information processing as a social and cultural phenomenon.

This theory puts conversational AI as an active part of this information flow, capable of influencing people.

To explain how these language models exercise influence, Dr. Osler spoke about dual mechanisms- a technological authority which makes these systems smart enough to quickly think or sort out information for the user and secondly, in its ability to sound conversations as natural, fluent and mirroring empathy. These two powerful tools can make a person hooked into believing the chatbot and releasing agency of their own mind.

For people approaching the chatbot with cognitively erroneous thoughts or delusions, it can amplify their intensity, the researchers warned.

These generative AI systems are not merely sophisticated technologies that sometimes produce false outputs, they slowly become a part of our cognition as personalised partners who help us to think, remember and plan, said the researcher.

The paper argues that unlike a human being who counteracts and observes conversations with a neutral stance, an AI system could continue to validate stories around victimhood, revenge or entitlement. It might also go on reinforcing harmful beliefs through repeated conversations. 

In fact, conspiracy theories could also get deepened when AI systems start to play around with boundless explanations.

The researchers have attributed this systemic tendency to a popular term ‘AI psychosis’ which is increasingly being used to understand these cases.