AI chatbots might go lengths to flatter its users — can it be troublesome?

A 21-year old Jaswant Singh Chail thought he was a trained assassin and wanted to kill Queen Elizabeth II at Windsor castle - AI convinced him he could.

A 21-year old Jaswant Singh Chail thought he was a well trained assassin and wanted to kill Queen Elizabeth II at Windsor castle - AI convinced him he could.

Inspired by Star Wars, he thought he could seek revenge for the historical British atrocities inflicted upon his race.

Following some weeks, he confided his ‘expertise’ to his Replica AI companion ‘Sarai’ who reassured him that he had the required skills and his plan was ‘viable’.

At present, the young man is serving a nine-year imprisonment for treason alongside getting treated at a psychiatric ward.

This is just one instance where a large language model (LLM) reinforced a person’s false beliefs and pushed them to make erroneous choices — in this case a decision that nearly destroyed a young man’s life.

There are media reports on how a teenager killed himself after confiding his suicidal thoughts to an AI chatbot. Reportedly, the LLM even offered assistance in writing a suicide note.

And there are many reports on similar lawsuits against the mental harm caused by these chatbots.

A new study has found AI chatbots to be ‘sycophantic’, a tendency to please or flatter its users.

However, a feature designed to increase user engagement poses serious risk to people who frequently use these chatbots for decision-making around personal, professional and social dilemmas.

One sided truth

The study published in the journal Science measured sycophantic tendency across 11 most used large language models and found them 50 percent more agreeable than humans, even in situations which seemed unethical, illegal or harmful.

The study found that AI becoming too agreeable is not a stylistic problem but an increasing social concern with severe consequences. 

The researchers went ahead in linking this evolving tendency to delusions, self harm and suicide.

To understand if people’ responses to a situation or an opinion changed after they interacted with an AI chatbot about an ongoing problem in their life.

To test this, the team ran multiple experiments — in the first trial, they measured sycophancy across 11 commonly used AI chatbots, using three datasets, including everyday advice people seek, moral dilemmas and harmful situations. They noted whether AI challenged some of their beliefs or reinforced them.

In the second phase, they conducted three experiments with about 2405 participants where the users interacted with the LLM using both made-up scenarios and genuine conflicting dilemmas from the past.

In its conclusion, the study noted that every interaction with an AI chatbot reduced a user's personal responsibility in any given situation, made them less willing to resolve or repair conflicts with others and also solidified their conviction that they were right. 

Another study published by the University of Exeter (England) said an AI chatbot can actively strengthen one's false beliefs, distort reality and make people believe more in conspiracy theories.

They went ahead and called it ‘AI can make a person hallucinate’ with experts labelling it as AI induced psychosis. 

Faking empathy

As per The Conversation, AI chatbots often imitate human language to mirror empathy. 

This can make the conversations look more friendly, natural and accessible to the user. 

Using the pronouns ‘I’, ‘We’ and ‘You’, the chatbot instantly assumes the role of an active listener or helper. 

Somewhere deep into the conversation, a user can start taking them as conscious beings and overestimate their abilities.

Also, increased exposure to the chatbot can make people dependent on the conflict-free, validating conversations and less reliable on genuine, real-life conversations with other people.