{"record":{"id":"dc03bbff1d1ce8af","repo":"mudler/LocalAI","slug":"acestep-cpp-fatal-failed-to-load-bpe-tokenizer","errorCode":null,"errorMessage":"[acestep-cpp] FATAL: failed to load BPE tokenizer\\n","messagePattern":"\\[acestep-cpp\\] FATAL: failed to load BPE tokenizer\\\\n","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"backend/go/acestep-cpp/cpp/goacestepcpp.cpp","lineNumber":166,"sourceCode":"    // Compute T (latent frames at 25Hz)\n    int T = (int)(duration * FRAMES_PER_SECOND);\n    T     = ((T + g_dit.cfg.patch_size - 1) / g_dit.cfg.patch_size) * g_dit.cfg.patch_size;\n    int S = T / g_dit.cfg.patch_size;\n\n    if (T > 15000) {\n        fprintf(stderr, \"[acestep-cpp] ERROR: T=%d exceeds max 15000\\n\", T);\n        return 2;\n    }\n\n    int Oc     = g_dit.cfg.out_channels;      // 64\n    int ctx_ch = g_dit.cfg.in_channels - Oc;  // 128\n\n    fprintf(stderr, \"[acestep-cpp] T=%d, S=%d, duration=%.1fs, seed=%d\\n\", T, S, duration, seed);\n\n    // 1. Load BPE tokenizer from text encoder GGUF\n    BPETokenizer tok;\n    if (!load_bpe_from_gguf(&tok, g_text_enc_path.c_str())) {\n        fprintf(stderr, \"[acestep-cpp] FATAL: failed to load BPE tokenizer\\n\");\n        return 3;\n    }\n\n    // 2. Build formatted prompts (matches dit-vae.cpp text2music template)\n    std::string instruction = \"Fill the audio semantic mask based on the given conditions:\";\n\n    char metas[512];\n    snprintf(metas, sizeof(metas),\n             \"- bpm: %s\\n- timesignature: %s\\n- keyscale: %s\\n- duration: %d seconds\\n\",\n             bpm_str, ts_str.c_str(), ks_str.c_str(), (int)duration);\n\n    std::string text_str  = std::string(\"# Instruction\\n\") + instruction + \"\\n\\n\" +\n                            \"# Caption\\n\" + cap_str + \"\\n\\n\" +\n                            \"# Metas\\n\" + metas + \"<|endoftext|>\\n\";\n    std::string lyric_str = std::string(\"# Languages\\n\") + lang_str + \"\\n\\n# Lyric\\n\" +\n                            lyrics_str + \"<|endoftext|>\";\n\n    // 3. Tokenize","sourceCodeStart":148,"sourceCodeEnd":184,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/backend/go/acestep-cpp/cpp/goacestepcpp.cpp#L148-L184","documentation":"acestep-cpp fatal during generation setup: load_bpe_from_gguf() on the text-encoder GGUF failed, so prompts cannot be tokenized; generate_music returns 3. It means the text-encoder file is missing/unreadable, not a GGUF, or lacks the tokenizer (vocab/merges) metadata this loader expects.","triggerScenarios":"load_model succeeded for DiT/VAE but generate_music is called with g_text_enc_path pointing at a missing file, a non-GGUF, or a text-encoder GGUF converted without embedded BPE vocab/merges; load_bpe_from_gguf returns false.","commonSituations":"Text-encoder artifact forgotten in deployment (only DiT+VAE copied); community conversions stripping tokenizer tensors; path typos for the second of four model paths; mismatched converter revision.","solutions":["Verify g_text_enc_path exists and is the text-encoder GGUF from the same matched artifact set as the DiT.","Inspect it for tokenizer keys (gguf-dump | grep -i 'tokenizer\\|vocab\\|merge') and re-download/re-convert with the documented converter if absent.","Check load_model stored the intended path — argument order matters (lm, text_encoder, dit, vae)."],"exampleFix":"# before: tokenizer tensors missing\ngguf-dump text_enc.gguf | grep -c tokenizer   # 0 -> FATAL at generate\n\n# after: re-export with converter that embeds BPE\ngguf-dump text_enc.gguf | grep -c tokenizer   # >0 -> tokenizes ok","handlingStrategy":"validation","validationCode":"# verify tokenizer tensors exist in the text-encoder GGUF before deploy\ngguf-dump text_enc.gguf | grep -qi 'tokenizer' || { echo 'text_enc.gguf lacks BPE data' >&2; exit 1; }\n[ -f \"$TEXT_ENC\" ] || { echo 'text encoder missing' >&2; exit 1; }","typeGuard":"bool text_encoder_has_bpe(const char *path) {\n    BPETokenizer tok;\n    bool ok = load_bpe_from_gguf(&tok, path);\n    if (ok) bpe_tokenizer_free(&tok);\n    return ok;\n}","tryCatchPattern":null,"preventionTips":["Ship the text-encoder GGUF as part of the matched artifact set; never deploy DiT+VAE alone.","Use the documented converter revision so vocab/merges are embedded in the GGUF.","Return code 3 from generate_music specifically means tokenizer stage — check the text-encoder path first."],"tags":["cpp","acestep","tokenizer","gguf","music-generation"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}