Zackriya-Solutions/meetily · error · ParakeetError

Missing <blk> token in vocabulary

Error message

Missing <blk> token in vocabulary

What it means

Parakeet's load_vocab reads vocab.txt from the model directory, parses lines of 'token id' pairs, and records the id of the special CTC blank token '<blk>'. If no line has token '<blk>', blank_idx stays None and an io::Error(InvalidData) is returned: CTC decoding is impossible without a blank index, so model construction aborts. The loader also replaces SentencePiece '▁' (U+2581) with spaces, confirming it expects the NeMo Parakeet vocab format.

Source

Thrown at frontend/src-tauri/src/parakeet_engine/model.rs:169

                let token = parts[0].to_string();
                if let Ok(id) = parts[1].parse::<usize>() {
                    if token == "<blk>" {
                        blank_idx = Some(id);
                    }
                    tokens_with_ids.push((token, id));
                    max_id = max_id.max(id);
                }
            }
        }

        // Create vocab vector with \u2581 replaced with space
        let mut vocab = vec![String::new(); max_id + 1];
        for (token, id) in tokens_with_ids {
            vocab[id] = token.replace('\u{2581}', " ");
        }

        let blank_idx = blank_idx.ok_or_else(|| {
            ParakeetError::Io(std::io::Error::new(
                std::io::ErrorKind::InvalidData,
                "Missing <blk> token in vocabulary",
            ))
        })? as i32;

        Ok((vocab, blank_idx))
    }

    pub fn preprocess(
        &mut self,
        waveforms: &ArrayViewD<f32>,
        waveforms_lens: &ArrayViewD<i64>,
    ) -> Result<(ArrayD<f32>, ArrayD<i64>), ParakeetError> {
        log::trace!("Running Parakeet preprocessor inference...");
        let inputs = inputs![
            "waveforms" => TensorRef::from_array_view(waveforms.view())?,
            "waveforms_lens" => TensorRef::from_array_view(waveforms_lens.view())?,
        ];

View on GitHub (pinned to 0281737d87)

Solutions

1. Re-download or re-extract the full parakeet model directory so vocab.txt matches the .onnx file's release
2. Inspect vocab.txt: confirm a line exactly matching '<blk> <number>' exists and lines are 'token id' pairs
3. If your vocab uses a different blank-token spelling, either patch vocab.txt to use '<blk>' or adapt the comparison in load_vocab
4. Verify the file is UTF-8 and not truncated (last line present, token count consistent with the model's output size)

Example fix

// vocab.txt (before) — blank token named differently
#blank 0
▁hello 1

// vocab.txt (after) — NeMo Parakeet convention
<blk> 0
▁hello 1

// Rust: validate vocab.txt before constructing the Parakeet model
fn vocab_has_blank(model_dir: &Path) -> bool {
    std::fs::read_to_string(model_dir.join("vocab.txt"))
        .map(|c| c.lines().any(|l| l.trim_end().split(' ').next() == Some("<blk>")))
        .unwrap_or(false)
}

// Rust: narrow an parsed vocab into a guaranteed-CTC-compatible one
struct CtcVocab { tokens: Vec<String>, blank_idx: i32 }
fn as_ctc_vocab(v: Vec<String>) -> Option<CtcVocab> {
    v.iter().position(|t| t == "<blk>")
        .map(|i| CtcVocab { tokens: v, blank_idx: i as i32 })
}

match ParakeetModel::new(&model_dir) {
    Err(ParakeetError::Io(ref e)) if e.to_string().contains("<blk>") => {
        eprintln!("vocab.txt is not a NeMo Parakeet vocab — re-extract the matching model assets");
        // re-download model bundle and retry once
    }
    other => other,
}

When it happens

Trigger: Loading a parakeet model directory whose vocab.txt comes from a different NeMo release that renamed the blank token; a truncated or empty vocab.txt; a hand-edited vocab where the '<blk> <id>' line was removed or the line format ('token id', space-separated) is violated so parsing skips it.

Common situations: Mixing model .onnx and vocab.txt assets from different parakeet versions; partial extraction of the model archive; upstream NeMo changing token conventions; passing a whisper-style vocab by mistake.

Related errors

• Parakeet transcription failed on segment {}: {}
• Whisper transcription failed on segment {}: {}
• Parakeet engine not initialized
• Parakeet transcription failed on segment {}: {}
• Failed to load Parakeet model {}: {}

AI-assisted analysis of Zackriya-Solutions/meetily@0281737d87 (2026-08-16). Data as JSON: /api/errors/8a1fd526c3a1ad5b. Report an issue: GitHub.