{"record":{"id":"8a1fd526c3a1ad5b","repo":"Zackriya-Solutions/meetily","slug":"missing-blk-token-in-vocabulary","errorCode":null,"errorMessage":"Missing <blk> token in vocabulary","messagePattern":"Missing <blk> token in vocabulary","errorType":"exception","errorClass":"ParakeetError","httpStatus":null,"severity":"error","filePath":"frontend/src-tauri/src/parakeet_engine/model.rs","lineNumber":169,"sourceCode":"                let token = parts[0].to_string();\n                if let Ok(id) = parts[1].parse::<usize>() {\n                    if token == \"<blk>\" {\n                        blank_idx = Some(id);\n                    }\n                    tokens_with_ids.push((token, id));\n                    max_id = max_id.max(id);\n                }\n            }\n        }\n\n        // Create vocab vector with \\u2581 replaced with space\n        let mut vocab = vec![String::new(); max_id + 1];\n        for (token, id) in tokens_with_ids {\n            vocab[id] = token.replace('\\u{2581}', \" \");\n        }\n\n        let blank_idx = blank_idx.ok_or_else(|| {\n            ParakeetError::Io(std::io::Error::new(\n                std::io::ErrorKind::InvalidData,\n                \"Missing <blk> token in vocabulary\",\n            ))\n        })? as i32;\n\n        Ok((vocab, blank_idx))\n    }\n\n    pub fn preprocess(\n        &mut self,\n        waveforms: &ArrayViewD<f32>,\n        waveforms_lens: &ArrayViewD<i64>,\n    ) -> Result<(ArrayD<f32>, ArrayD<i64>), ParakeetError> {\n        log::trace!(\"Running Parakeet preprocessor inference...\");\n        let inputs = inputs![\n            \"waveforms\" => TensorRef::from_array_view(waveforms.view())?,\n            \"waveforms_lens\" => TensorRef::from_array_view(waveforms_lens.view())?,\n        ];","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/Zackriya-Solutions/meetily/blob/0281737d87d26352fb0adc78c8c0975f691b23d1/frontend/src-tauri/src/parakeet_engine/model.rs#L151-L187","documentation":"Parakeet's load_vocab reads vocab.txt from the model directory, parses lines of 'token id' pairs, and records the id of the special CTC blank token '<blk>'. If no line has token '<blk>', blank_idx stays None and an io::Error(InvalidData) is returned: CTC decoding is impossible without a blank index, so model construction aborts. The loader also replaces SentencePiece '▁' (U+2581) with spaces, confirming it expects the NeMo Parakeet vocab format.","triggerScenarios":"Loading a parakeet model directory whose vocab.txt comes from a different NeMo release that renamed the blank token; a truncated or empty vocab.txt; a hand-edited vocab where the '<blk> <id>' line was removed or the line format ('token id', space-separated) is violated so parsing skips it.","commonSituations":"Mixing model .onnx and vocab.txt assets from different parakeet versions; partial extraction of the model archive; upstream NeMo changing token conventions; passing a whisper-style vocab by mistake.","solutions":["Re-download or re-extract the full parakeet model directory so vocab.txt matches the .onnx file's release","Inspect vocab.txt: confirm a line exactly matching '<blk> <number>' exists and lines are 'token id' pairs","If your vocab uses a different blank-token spelling, either patch vocab.txt to use '<blk>' or adapt the comparison in load_vocab","Verify the file is UTF-8 and not truncated (last line present, token count consistent with the model's output size)"],"exampleFix":"// vocab.txt (before) — blank token named differently\n#blank 0\n▁hello 1\n\n// vocab.txt (after) — NeMo Parakeet convention\n<blk> 0\n▁hello 1","handlingStrategy":"validation","validationCode":"// Rust: validate vocab.txt before constructing the Parakeet model\nfn vocab_has_blank(model_dir: &Path) -> bool {\n    std::fs::read_to_string(model_dir.join(\"vocab.txt\"))\n        .map(|c| c.lines().any(|l| l.trim_end().split(' ').next() == Some(\"<blk>\")))\n        .unwrap_or(false)\n}","typeGuard":"// Rust: narrow an parsed vocab into a guaranteed-CTC-compatible one\nstruct CtcVocab { tokens: Vec<String>, blank_idx: i32 }\nfn as_ctc_vocab(v: Vec<String>) -> Option<CtcVocab> {\n    v.iter().position(|t| t == \"<blk>\")\n        .map(|i| CtcVocab { tokens: v, blank_idx: i as i32 })\n}","tryCatchPattern":"match ParakeetModel::new(&model_dir) {\n    Err(ParakeetError::Io(ref e)) if e.to_string().contains(\"<blk>\") => {\n        eprintln!(\"vocab.txt is not a NeMo Parakeet vocab — re-extract the matching model assets\");\n        // re-download model bundle and retry once\n    }\n    other => other,\n}","preventionTips":["Always ship/extract the .onnx and vocab.txt from the same parakeet release bundle","Never hand-edit vocab.txt without keeping an exact '<blk> <id>' line","Smoke-test model loading (which parses the vocab) right after extraction, before starting capture"],"tags":["parakeet","nemo","ctc","vocabulary","rust","transcription"],"backgroundTag":"tokenizer-vocab-mismatch","analyzedSha":"0281737d87d26352fb0adc78c8c0975f691b23d1","analyzedAt":"2026-08-16T20:57:52.567Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}