Google DeepMind unveils sign-language-to-text AI, now shipping in Pixel 11's Gboard and Live Transcribe

Google DeepMind has introduced SL2T (sign-language-to-text), a translation model that converts sign language into text, and is bringing it to consumer devices for the first time. The model powers sign-to-text dictation in Gboard and Live Transcribe on the Pixel 11, starting with American Sign Language (ASL) to English; more devices and languages are to follow. Deaf and hard-of-hearing users can now sign to their phone anywhere they would normally type.

There are more than 200 sign languages worldwide, used by an estimated 70 million Deaf and hard-of-hearing people. Sign language processing has lagged behind spoken language processing for two reasons, DeepMind said. Sign languages are independent languages with their own grammars and lexicons, so translation is not a straightforward sound-to-text mapping. And the model has to accurately track simultaneous movements of the hands, arms, torso, head and face, which is a demanding computer vision task.

SL2T was trained on more than 100,000 hours of data across more than 50 sign languages, roughly a quarter of it in ASL. Training jointly across many languages, dialects and proficiency levels lets the model learn shared underlying structures, which outperformed single-language models in DeepMind's experiments. Rather than translating from raw camera footage, an on-device model called MediaPipe Holistic tracks pose landmarks on the signer's body and only these coordinates are sent to the server; the original video is discarded immediately, to protect user privacy.

SL2T bypasses the intermediate annotations known as 'glosses' that earlier systems widely used, and translates the landmark sequence directly into text. Glosses fail to capture non-manual markers and spatial constructions, DeepMind said. On the FLEURS-ASL benchmark for ASL-to-English translation, SL2T achieves a zero-shot score of 70 BLEURT, which the team says is significantly higher than any previously reported result.

The team also worked on practical issues — minimising streaming latency, avoiding hallucinated output on non-signing input, ensuring fair performance for the roughly 10 percent of signers who are left-handed, and supporting one-handed signing that a user might do while holding a phone in the other hand.