{"record":{"id":"0bbe9c536709468d","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-model-enables-byte-fallback-but","errorCode":null,"errorMessage":"The tokenizer.json model enables byte_fallback but does not contain a contiguous <0x00>..<0xFF> byte-piece block required to represent it.","messagePattern":"The tokenizer\\.json model enables byte_fallback but does not contain a contiguous <0x00>\\.\\.<0xFF> byte-piece block required to represent it\\.","errorType":"exception","errorClass":"InvalidDataException","httpStatus":null,"severity":"critical","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs","lineNumber":220,"sourceCode":"                }\n                else\n                {\n                    _vocabReverse[i] = (piece, score, ModelProto.Types.SentencePiece.Types.Type.Normal);\n                    _vocab.Add(piece, i);\n                    _minScore = Math.Min(_minScore, score);\n                    _maxScore = Math.Max(_maxScore, score);\n                }\n            }\n\n            if (ByteFallback)\n            {\n                // Byte fallback requires a contiguous block of the 256 byte pieces <0x00>..<0xFF>; encode/decode map a\n                // byte value to ByteCodeToIdOffset + value. Validate it (the proto path relies on the same layout) and\n                // set MaxByteId from <0xFF> so byte ids are recognized on decode, rather than misencoding silently.\n                ByteCodeToIdOffset = _vocab.TryGetValue(\"<0x00>\", out int id) ? id : MaxByteId;\n                if (!_vocab.ContainsKey(\"<0x00>\") || !_vocab.TryGetValue(\"<0xFF>\", out int maxByteId) || maxByteId - ByteCodeToIdOffset != 0xFF)\n                {\n                    throw new InvalidDataException(\"The tokenizer.json model enables byte_fallback but does not contain a contiguous <0x00>..<0xFF> byte-piece block required to represent it.\");\n                }\n\n                MaxByteId = maxByteId;\n                OneByteUtf8EncodingMaxId = ByteCodeToIdOffset + 0x7F;\n                MaxIdByteFallbackId = ByteCodeToIdOffset + 0xFF;\n            }\n            // When byte fallback is disabled the byte offsets stay at 0 so decode treats no ids as byte pieces, even\n            // if the vocab happens to contain <0xNN> entries (otherwise normal low ids would be dropped as bytes).\n\n            _trie = new DoubleArrayTrie(_vocab);\n\n            // Re-insert the unknown token into the vocab maps after the trie is built so it maps like a regular token.\n            // A negative unkId means the model has no unknown token (byte fallback covers OOV), so there is nothing to\n            // re-insert in that case.\n            if (unkId >= 0)\n            {\n                string unkToken = pieces[unkId].Piece;\n                _vocab[unkToken] = unkId;","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs#L202-L238","documentation":"Byte fallback in SentencePiece requires the vocabulary to contain the 256 byte pieces <0x00> through <0xFF> as one contiguous id block, because encode/decode maps byte value v to ByteCodeToIdOffset + v. If the tokenizer.json enables byte_fallback but the pieces are missing or not contiguous (maxByteId - offset != 0xFF), the model throws InvalidDataException instead of misencoding arbitrary bytes.","triggerScenarios":"Constructing SentencePieceUnigramModel from a tokenizer.json with model.byte_fallback == true where <0x00> or <0xFF> is absent, or where their ids differ by more than 0xFF.","commonSituations":"Tokenizers whose byte pieces were pruned during vocabulary filtering; models trained without byte fallback but with the flag turned on in exported config; conversion tools that reordered or dropped special pieces.","solutions":["Regenerate the tokenizer with byte_fallback training enabled so the full <0x00>..<0xFF> block is emitted contiguously.","Disable byte_fallback in the tokenizer.json model config if the vocabulary does not actually contain the byte pieces.","Check the vocabulary with Python tokenizers/sentencepiece to verify ids of <0x00> and <0xFF> differ by exactly 255, and fix any reordering of the byte block.","Re-export the model from the original training checkpoint rather than editing pieces manually."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// C# — check byte_fallback consistency in tokenizer.json before loading\nvar model = json.RootElement.GetProperty(\"model\");\nbool byteFallback = model.TryGetProperty(\"byte_fallback\", out var bf) && bf.GetBoolean();\nif (byteFallback)\n{\n    var vocab = model.GetProperty(\"vocab\").EnumerateObject();\n    bool hasFirst = false, hasLast = false;\n    foreach (var p in vocab)\n    {\n        if (p.Name == \"<0x00>\") hasFirst = true;\n        if (p.Name == \"<0xFF>\") hasLast = true;\n    }\n    if (!hasFirst || !hasLast)\n        throw new InvalidDataException(\"byte_fallback=true requires contiguous <0x00>..<0xFF> pieces.\");\n}","typeGuard":"static bool BytePiecesContiguous(int id00, int idFF) => idFF - id00 == 0xFF;","tryCatchPattern":"try { var model = new SentencePieceUnigramModel(...); }\ncatch (InvalidDataException ex) { /* fix byte_fallback flag or regenerate vocab */ }","preventionTips":["Only enable byte_fallback when training actually produced the 256 byte pieces.","Never prune special/byte pieces from a byte_fallback vocabulary.","Keep byte pieces in their original contiguous id order in exports.","Validate byte-piece coverage in Python before shipping the tokenizer."],"tags":["sentencepiece","byte-fallback","vocabulary","model-loading"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}