{"record":{"id":"27f98d9f7a1ebea4","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-model-has-a-null-unk-id-but-d","errorCode":null,"errorMessage":"The tokenizer.json model has a null 'unk_id' but does not enable 'byte_fallback'; a Unigram model without an unknown token is only supported when byte_fallback is enabled.","messagePattern":"The tokenizer\\.json model has a null 'unk_id' but does not enable 'byte_fallback'; a Unigram model without an unknown token is only supported when byte_fallback is enabled\\.","errorType":"validation","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":656,"sourceCode":"                }\n\n                string? piece = entry[0].GetString();\n                if (piece is null)\n                {\n                    throw new InvalidDataException(\"A piece string in 'model.vocab' is null.\");\n                }\n\n                vocab.Add((piece, entry[1].GetSingle()));\n            }\n\n            if (unkIsNull)\n            {\n                // Without an unknown token the only way to represent out-of-vocabulary input is byte fallback; a model\n                // with neither cannot encode OOV text, so reject that combination up front rather than emitting an\n                // invalid token id at encode time.\n                if (!byteFallback)\n                {\n                    throw new NotSupportedException(\"The tokenizer.json model has a null 'unk_id' but does not enable 'byte_fallback'; a Unigram model without an unknown token is only supported when byte_fallback is enabled.\");\n                }\n            }\n            else if (unkId < 0 || unkId >= vocab.Count)\n            {\n                throw new InvalidDataException($\"The tokenizer.json model 'unk_id' ({unkId}) is out of range for a vocabulary of {vocab.Count} pieces.\");\n            }\n\n            // Extract normalizer settings\n            byte[]? precompiledCharsMap = null;\n            bool addDummyPrefix = true;\n            // HF tokenizer.json has no remove_extra_whitespaces flag; SpmConverter encodes that behavior as\n            // explicit normalizer steps (a right-Strip plus a Replace collapsing runs of spaces). Deduce it from\n            // those steps, defaulting to false when absent to match the HF fast-tokenizer runtime.\n            bool removeExtraWhitespaces = false;\n            // When the normalizer has content-modifying steps that the charsmap + removeExtraWhitespaces\n            // approximation cannot represent (per-character Replace, Lowercase, Unicode normalization, ...),\n            // apply the full normalizer chain (charsmap included) before the Metaspace pass instead.\n            SentencePieceNormalizationStep? chainNormalizer = null;","sourceCodeStart":638,"sourceCodeEnd":674,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L638-L674","documentation":"When unk_id is null the model has no unknown token, so out-of-vocabulary text can only be encoded via byte fallback. If byte_fallback is not enabled, no valid encoding exists for OOV input, so the loader throws NotSupportedException up front rather than producing invalid token ids at encode time.","triggerScenarios":"CreateFromTokenizerJson where model.unk_id is null and model.byte_fallback is absent or false.","commonSituations":"Custom-trained SentencePiece models trained with --hard_vocab_limit and no unk piece and without byte fallback; hand-edited tokenizer.json setting unk_id to null; stripping <unk> from the vocab without enabling byte_fallback.","solutions":["Set byte_fallback: true in the model section (and include the <0x00>-<0xFF> byte pieces in the vocab, as the original training did).","Alternatively give the model a real unknown token: set unk_id to the index of an '<unk>' piece present in vocab.","Retrain/re-export the tokenizer with unk_id defined if byte fallback is not desired."],"exampleFix":"// before\n\"model\": { \"type\": \"Unigram\", \"unk_id\": null, \"byte_fallback\": false, \"vocab\": [...] }\n// after\n\"model\": { \"type\": \"Unigram\", \"unk_id\": null, \"byte_fallback\": true, \"vocab\": [[\"<0x00>\", ...], ..., [\"<0xFF>\", ...], ...] }","handlingStrategy":"validation","validationCode":"var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty(\"model\");\nbool unkNull = model.GetProperty(\"unk_id\").ValueKind == JsonValueKind.Null;\nbool bf = model.TryGetProperty(\"byte_fallback\", out var b) && b.ValueKind == JsonValueKind.True;\nif (unkNull && !bf)\n    throw new InvalidOperationException(\"null unk_id requires byte_fallback: true (with byte pieces in vocab).\");","typeGuard":null,"tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (NotSupportedException ex) when (ex.Message.Contains(\"byte_fallback\")) { /* choose a tokenizer with an unk token or byte fallback */ }","preventionTips":["Only use unk_id: null with models trained with byte fallback enabled.","Keep the <0x00>..<0xFF> byte pieces in vocab when byte_fallback is true.","Match training-time SentencePiece flags when exporting tokenizer.json."],"tags":["tokenizer","configuration","unsupported-combination"],"backgroundTag":"conflicting-config-options","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}