Adalat AI discovered that OpenAI’s Whisper speech recognition model truncates Indic language transcripts after approximately six seconds, ending with a Unicode replacement character. The root cause is that Whisper’s 448-token capacity is split in half by design, leaving only 224 tokens for new output, while Indic scripts require roughly three tokens per character due to minimal representation in the tokenizer’s training data. The team implemented fixes including shorter audio segmentation windows and removing the static token-budget halving in their CTranslate2 fork, improving error rates by up to 7 percentage points on Malayalam test sets.
