Multimodal · Accessibility
Sign Bridge
Speech, text and video translated into sign language by a 3D avatar.
Sign Bridge translates speech, text and video into sign language rendered by a 3D avatar, treating sign as a language to translate into rather than a caption to display. It is two codebases: a FastAPI backend for transcription and NLP, and a Unity client that drives the avatar.
Problem
Sign language is not a word-for-word encoding of speech. A literal transcription produces something a deaf user cannot read, so translation has to happen at the level of meaning.
Objectives
- Accept speech, text and video without three separate implementations.
- Translate at the level of meaning rather than transcribing word for word.
- Render output as animation that can grow without new recordings.
Implementation
- A FastAPI service normalising all three input modalities into one internal representation before translation begins.
- A BERT-based NLP stage, supported by NLTK preprocessing, mapping input to sign gloss.
- OpenPose pose estimation providing the keypoint vocabulary that drives avatar motion.
- A Unity 3D avatar animating the gloss sequence as the user-facing output.
Architecture
- speech
- text
- video
- Whisper STT
- BERT
- NLTK
- sign gloss
- OpenPose keypoints
- Unity 3D avatar
Technologies
- FastAPI
- Whisper
- BERT
- Transformers
- OpenPose
- Unity
- NLTK
- PostgreSQL
- Pandas
Challenges & solutions
Challenge
Sign language grammar does not follow spoken word order, so literal transcription produces unreadable output.
Solution
Introduced gloss as an intermediate representation, making translation an explicit stage rather than an implicit one.
Challenge
Three input modalities risk becoming three divergent codepaths.
Solution
Converged all inputs to a single representation early, so the translation and rendering stages have exactly one contract to satisfy.
Results
- One FastAPI service handles speech, text and video through a single translation path, so the three entry points cannot drift apart.
- Translation happens at the level of gloss rather than word order, so the output follows sign grammar instead of transcribed speech.
- Output is a rendered 3D avatar rather than a video lookup, so the vocabulary can grow without recording new footage.
- The vocabulary covers roughly 32,000 sentences and 4,000 individual words, plus the alphabet and numbers for spelling anything outside it.
Lessons learned
- An intermediate representation is what turns a demo into a system: it gives every stage a contract instead of an assumption.
- Accessibility work rewards talking to the people who will use it more than it rewards model choice.