Humour upgraded: AAC innovation helps non-verbal users express emotions

Augmented Alternative Communication systems are adding humour with AI, enabling non-verbal users to express emotions, jokes, and personality in real-time, writes Vijayalakshmi Sridhar.

Prithvi (name changed), a 13-year-old autistic child, has broken his dad’s laptop. His parents are mad at him for being careless. While they angrily go on telling him about how and why he needs to be more careful, Prithvi uses his Augmented Alternative Communication (AAC) device to tell them he needs a hug immediately. The surprised parents instantly calm down and scoop Prithvi up in a bear hug. Thanks to his communication partner’s efforts, Prithvi knew how to use his AAC device well.

For anyone with speech impairments and impediments because of illness, injury or lifelong conditions, hi-tech aided-AAC devices act as their voice. These systems, installable as an application on iPhones, desktops, or iPads, comprise a collection of pictures on an easily accessible grid with matching sounds. For those who move from tapping to typing, there is an attached keyboard, and the text-to-speech tool speaks the typed-out phrases.

In the hi-tech category, there are two made-in-India AAC device systems used dynamically for their regional language options.

Jellow, a free complete communication system, conceived by researchers at the IDC School of Design, IIT Bombay and Avaz AAC, developed by researchers at IIT-Madras, was first introduced as a tablet computer and then remodelled as an app with both free and paid options. Both can be installed across Android, iOS and Kindle devices. “Apart from Jellow and Avaz’s free versions for children with intellectual disability, cerebral palsy, those who are mute on the spectrum, adults who have neuro-linguistic issues like dementia, and those afflicted by stroke, we also use low-tech cards and cardboard, where we print the pictures. For the majority, the devices are helpful to establish basic communication abilities. We are yet to move on to higher emotions or more pragmatic use of language,” says Dr Namita Joshi, Hon General Secretary (Ad-hoc), Indian Speech-Language and Hearing Association (ISHA) Office, Mysuru.

Holistic learning

In addition to a standard menu, AAC system developers keep adding options of words to make the device customisable for each user. Special family or peer group dialect words, funny reference words and even words from specific interest areas of the child are added to the device.

The AAC’s voice output is a source of both reassurance and empowerment to the child. “The child’s journey into and through language begins through the AAC system,” says board-certified behaviour analyst and special educator Kamalathara Srikanth.

Early introduction of AAC for the non-verbals is a big plus, as it facilitates positive communication and language skills. Once the barriers are broken, like in her daughter Vaanya’s case, it has been possible to improve her learning in more ways than one, says Keerthana Kolli, product manager at Avaz Inc. Keerthana introduced Avaz to her daughter Vaanya when she was four. After a year or so, one summer, Keerthana’s mom had come visiting, and she had spent a lot of time with Vaanya. After a week, her grandma left, and Vaanya came home from school to learn that her grandma was not around in the house. Surely Vaanya missed her grandma, but she didn’t know how to articulate the feeling. Later, when she was watching a Baby Shark movie, Vaanya typed ‘grandma shark” on her AAC system. To Keerthana, her daughter’s intuitive action came as a huge surprise. “We are so happy she processed the feeling and expressed it. We were in disbelief that she spelt and typed ‘grandma,’ all by herself,” says Keerthana.

“There is a general assumption that AAC users do not have much to say since they can’t. On the contrary, they have a lot to say,” says Narayanan Ramakrishnan, CEO, Avaz Inc. So, in 2023, going a step ahead, Avaz introduced Expressive Tones, a tonal improvisation to the usual text-to-speech on the device. Using this button, the users will hear the typed-out phrases with corresponding emotional intonation for angry, sad, excited, curious, sarcastic, and scared words.

AI, the non-verbal partner

AI and AAC Technology have been a great aid for AAC upgrades so far. Now, Artificial Intelligence has taken the possibilities several notches up. A lot of big-leap value-adds in AAC have been made possible by AI. AI can be an AAC user’s personal search and vocabulary-building assistant. 

For those with language processing and expression difficulties, instead of a manual addition, AI can search for words and build a complete vocabulary menu around their topics of interest.

The LLMs can create personalities so that users can select the one that is close to their own and let the AI build word boards for them.

AI can accomplish ‘Just in time customisation’ too, for example, if they are in McDonald’s and the communication partner needs a set of words and visuals on McDonald’s, AI creates it in a jiffy.

A recent study by researchers at Cornell Tech and the Cornell Ann S Bowers College of Computing and Information Science, New York, attempts to take this effort to the next step.

The paper has presented thoughts on how AI interfaces can help non-verbal people to express humour. The study titled, “Why so serious? Exploring timely, humorous comments in AAC through AI-powered interfaces,” is on how AAC users can crack jokes in their voice and style, in real-time. The four AI-assisted interfaces, viz. Full-auto, Keywords, Context Bubble Selection, and Wizard could aid users in either cutting quick jokes or slow, carefully-crafted, and edited ones.

But, for the non-verbal, especially autistic and low-cognition individuals, humour is a complex, hard-to-understand emotion. Though the Avaz app has a humour button, it is intended for the adult market. “The younger children enjoy the pre-programmed jokes. But expressing humour through words is still not there for the older people,” adds Keerthana.

Bridging the gap

Though several Indian and international institutions are engaged in advocacy and inclusion for AAC use, there is still a gap in accessibility because of a lack of awareness about their conditions and the means to own the devices. Adding to it are apprehensions that gadgets would delay speech development. Experts insist that AAC is a two-way communication system that cares about users’ selections. 

At the end of the day, effective use of the device and the users’ progress depend entirely upon the communication partner. ISHA claims that there are only 5,000 trained and registered Speech Language Pathologists (SLP) for over 50 million people needing communication assistance in India, with most of the communication partners being the parents. who guide the non-verbal child to communicate in a culturally-appropriate and contextually-appropriate manner. “The dearth is the real challenge,” says Dr Joshi.