AI goes haywire, invents its own policy after users get mysteriously logged out

AI startup Anysphere and its AI-powered software coding assistant, Cursor have landed in a controversy with its chatbot running rogue and giving made-up explanations after users were strangely logged out of the software.

According to a report by Fotune.com, a Cursor user posted on Hacker News and Reddit that customers had started getting mysteriously logged out when switching between devices. When the users contacted the organisation through customer support, they received an emailed response from “Sam”, that the logouts were “expected behavior” under a new login policy. Sam is actually an AI-powered bot behind the email support.

However, what came as a twist in the tale was that their was no new login policy and the bot was actually hallucinating.

The news spread rapidly in the developer community and users started cancelling their subscriptions. Some people also complained about the lack of transparency in the working of the startup.

Later, co-founder Michael Truell acknowledged the problem and posted on Reddit that the company was investigating a bug that logged users out.

Admitting the “incorrect response from a front-line AI support bot”, he apologised and wrote, “Apologies about the confusion here."

The incident has once again started a debate on excessive dependence on AI and controlling its behaviour.

Technology experts took to LinkedIn to express their views on the issue. "Cursor...just landed itself in a viral hot mess because it failed to tell users that its customer support "person" Sam is actually a hallucinating bot,” Cassie Kozyrkov, an AI advisor and Google’s former chief decision scientist, wrote in a LinkedIn post.

“This mess could have been avoided if leaders understood that (1) AI makes mistakes, (2) AI can't take responsibility for those mistakes (so it falls on you), and (3) users hate being tricked by a machine posing as a human,” she added.