AI-Based AAC with Next-Word Suggestions and Multi-Word Sentence Completion Using Personal Data and Context

Abstract

We present a new Augmentative and Alternative Communication (AAC) app designed for patients with anomic aphasia and older adults experiencing word-recall difficulties. Although there has been significant recent progress in AAC systems, users still face difficulties expressing complex intentions precisely in sentences of adequate length [1]. This primary challenge stems from the trade-off between vocabulary size and the time required to search for a word. Our app addresses this by incorporating next-core-word suggestions based on syntactic structures [2] alongside multi-word sentence completion powered by large language models (LLMs). Personalizing these language models with the user’s past conversational data and current situational context significantly improves word-suggestion accuracy and reduces search time.

Following Chomsky’s seminal work and subsequent research on syntactic structures, the essential components of a formal sentence comprise a subject and a verb, with the optional addition of objects, complements, and/or prepositional phrases. In the majority of conversational cases, AAC users structure sentences around the subject “I,” which can often be assumed and skipped. Therefore, we designed the word-suggestion component of the user interface into three color-coded categories: Verbs, Objects, and Others. The Others category includes complements, personal pronouns, and the core words (excluding prepositions) of prepositional phrases. At the start of a session, the app displays several likely words based on usage from previous sessions within each category. The user can either select one of these suggested words or initiate a two-level, tree-structured word search based on meaning.

Selecting a word in any category automatically triggers the next-word suggestion routine, which displays subsequent core-word suggestions across the remaining two categories from the app’s vocabulary list. Although LLMs were initially considered for this next-word suggestion task, we found that simple bigram and trigram models perform better and faster given a limited vocabulary and smaller personalized datasets. We developed specialized algorithms to selectively utilize parts of these bigrams and trigrams based on parts of speech (POS) for each of the three categories. For example, when a Verb is selected, subsequent word suggestions are restricted exclusively to the Objects and Others categories.

After selecting a sufficient number of words through suggestions and searches, the user chooses to form one of three sentence types: an affirmative sentence, a negative sentence, or a question. Because a limited set of keywords can result in ambiguity, an LLM proposes three likely candidate sentences for the user to select from before the final text is spoken via a text-to-speech (TTS) engine [3].

The app was evaluated by 11 individuals with word-recall difficulties. In approximately 80% of the test cases, sentences consisting of 3 to 4 words were successfully constructed using only 4 to 7 clicks. The time required to form a sentence was under 20 seconds for about 50% of the sentences, and under 40 seconds for approximately 70% of all cases. Users provided highly favorable feedback, averaging a score of 4.0 on a 5-point Likert scale for both the usefulness and appropriateness of the generated dialogue sentences.

References

[1] H. Curtis, T. Neate, C.V. Gonzalez, “State of the Art in AAC: A Systematic Review and Taxonomy,” ASSETS ’22, Oct. 2022, Athens, Greece.

[2] N. Chomsky, Syntactic Structures, Mouton & Co., 1957.

[3] Y. Lee, A. Rabiee, S.Y. Lee, “Emotional End-to-End Neural Speech synthesizer,” arXiv:1711.05447, 2017 (NIPS ML4Audio workshop).