A Multimodal, On-Device AI System for Real-Time, Context-Aware Vocabulary Suggestion on AAC Displays

Abstract

For many people who use aided AAC, the hardest part of communication is not knowing what to say. It is reaching the words fast enough for the moment to be there still. Most symbol-based systems present the same page organization regardless of setting, so a student who wants to answer a question in science class must navigate several levels of menus while the conversation moves on without them. The consequences of this friction are well documented: an estimated 4–5 million Americans cannot reliably meet daily communication needs through speech, yet 30–35% of speech-generating devices are abandoned within a year, with context-mismatched vocabulary a frequently named cause.

Dedicated devices cost $4,000–$12,000 and face reimbursement cycles of 3–18 months, and AAC users typically communicate at roughly 10% of conversational speed. The field does not only need better devices; it needs devices that keep up with the moment, at a price schools and families can actually reach.

VoiceLink addresses this timing problem by making the AAC display aware of its context while deliberately leaving the user’s core vocabulary untouched. The system has two parts. The first is a small wearable — a camera and microphone housed in a 3D-printed lanyard, built on an approximately $15 microcontroller — chosen over glasses or headsets because many users cannot tolerate head-worn hardware. The second is software running entirely on a consumer tablet the family or school already owns. The wearable streams to the tablet over a private, local network; no internet connection or external server is involved at any point.

On the tablet, three models run fully on-device. A lightweight vision model identifies the setting and salient objects in view. A speech recognition model transcribes the surrounding conversation in real time. A distilled language model then fuses both streams with a personal context profile — a short form that a parent, teacher, or speech-language pathologist completes once, describing the user’s current classroom topics, routines, sensory triggers, and communication preferences. The result is a small, continuously updated set of suggested words and phrases that fit the current situation: if the class is discussing photosynthesis, the relevant vocabulary is already on screen when the lesson begins.

Getting the suggestions right required more than raw prediction. Early versions repeatedly surfaced near-duplicate hedge phrases (“I don’t know” and its paraphrases) at the expense of substantive answers. We addressed this by semantically deduplicating the vocabulary and replacing simple top-k selection with a diversity-aware ranking strategy that guarantees room for genuine answers; redundant hedge suggestions in the top seven fell from 2.107 to 0.012 per query. To fit modest hardware, the language model was quantized to INT8, compressing it 2.7-fold while preserving a 0.950 mean overlap with the full model’s top-six rankings, and quantized on-device transcription outperformed a widely used optimized baseline (0.35 s versus 0.42 s per utterance). Real-time, context-aware suggestion is achievable on affordable, already-owned hardware.

A central design decision, shaped by feedback from speech-language pathologists, is that suggestions occupy a dedicated region of the display while core vocabulary remains fixed in place. Motor planning is one of the strongest assets an experienced AAC user has, and any system that rearranges familiar buttons risks doing more harm than good. VoiceLink only ever changes the suggestion strip. For communication partners, the same design reduces programming burden: rather than reprogramming boards mid-lesson, a teacher enters the week’s topics once.

Because the system involves a camera and microphone worn by a child, privacy is architectural rather than promissory: all inference is local, audio and video are held only in ephemeral buffers with no persistent recording, a visible indicator light shows when the camera is active, a physical button cuts recording instantly, and the personal context store never leaves the device. We will argue that on-device processing should be the default standard forAI-supported AAC, particularly for minors and school deployment, where cloud-based tools raise consent and data-governance questions many districts cannot answer.

VoiceLink was developed with input from families of AAC users, educators, and speech- language pathologists, and reviewed with faculty mentors at Emory University and Duke  University. Our presentation covers the motivation, the multimodal pipeline, the reasoning behind our core design constraints, our quantitative system results, and our roadmap toward structured user evaluation — along with the question we most want the AAC community’s guidance on: what context-aware suggestion should and should not do, and who should control it.