dotnet/machinelearning · critical · InvalidDataException

The tokenizer.json model enables byte_fallback but does not

Error message

The tokenizer.json model enables byte_fallback but does not contain a contiguous <0x00>..<0xFF> byte-piece block required to represent it.

What it means

Byte fallback in SentencePiece requires the vocabulary to contain the 256 byte pieces <0x00> through <0xFF> as one contiguous id block, because encode/decode maps byte value v to ByteCodeToIdOffset + v. If the tokenizer.json enables byte_fallback but the pieces are missing or not contiguous (maxByteId - offset != 0xFF), the model throws InvalidDataException instead of misencoding arbitrary bytes.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs:220

                }
                else
                {
                    _vocabReverse[i] = (piece, score, ModelProto.Types.SentencePiece.Types.Type.Normal);
                    _vocab.Add(piece, i);
                    _minScore = Math.Min(_minScore, score);
                    _maxScore = Math.Max(_maxScore, score);
                }
            }

            if (ByteFallback)
            {
                // Byte fallback requires a contiguous block of the 256 byte pieces <0x00>..<0xFF>; encode/decode map a
                // byte value to ByteCodeToIdOffset + value. Validate it (the proto path relies on the same layout) and
                // set MaxByteId from <0xFF> so byte ids are recognized on decode, rather than misencoding silently.
                ByteCodeToIdOffset = _vocab.TryGetValue("<0x00>", out int id) ? id : MaxByteId;
                if (!_vocab.ContainsKey("<0x00>") || !_vocab.TryGetValue("<0xFF>", out int maxByteId) || maxByteId - ByteCodeToIdOffset != 0xFF)
                {
                    throw new InvalidDataException("The tokenizer.json model enables byte_fallback but does not contain a contiguous <0x00>..<0xFF> byte-piece block required to represent it.");
                }

                MaxByteId = maxByteId;
                OneByteUtf8EncodingMaxId = ByteCodeToIdOffset + 0x7F;
                MaxIdByteFallbackId = ByteCodeToIdOffset + 0xFF;
            }
            // When byte fallback is disabled the byte offsets stay at 0 so decode treats no ids as byte pieces, even
            // if the vocab happens to contain <0xNN> entries (otherwise normal low ids would be dropped as bytes).

            _trie = new DoubleArrayTrie(_vocab);

            // Re-insert the unknown token into the vocab maps after the trie is built so it maps like a regular token.
            // A negative unkId means the model has no unknown token (byte fallback covers OOV), so there is nothing to
            // re-insert in that case.
            if (unkId >= 0)
            {
                string unkToken = pieces[unkId].Piece;
                _vocab[unkToken] = unkId;

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Regenerate the tokenizer with byte_fallback training enabled so the full <0x00>..<0xFF> block is emitted contiguously.
2. Disable byte_fallback in the tokenizer.json model config if the vocabulary does not actually contain the byte pieces.
3. Check the vocabulary with Python tokenizers/sentencepiece to verify ids of <0x00> and <0xFF> differ by exactly 255, and fix any reordering of the byte block.
4. Re-export the model from the original training checkpoint rather than editing pieces manually.

// C# — check byte_fallback consistency in tokenizer.json before loading
var model = json.RootElement.GetProperty("model");
bool byteFallback = model.TryGetProperty("byte_fallback", out var bf) && bf.GetBoolean();
if (byteFallback)
{
    var vocab = model.GetProperty("vocab").EnumerateObject();
    bool hasFirst = false, hasLast = false;
    foreach (var p in vocab)
    {
        if (p.Name == "<0x00>") hasFirst = true;
        if (p.Name == "<0xFF>") hasLast = true;
    }
    if (!hasFirst || !hasLast)
        throw new InvalidDataException("byte_fallback=true requires contiguous <0x00>..<0xFF> pieces.");
}

static bool BytePiecesContiguous(int id00, int idFF) => idFF - id00 == 0xFF;

try { var model = new SentencePieceUnigramModel(...); }
catch (InvalidDataException ex) { /* fix byte_fallback flag or regenerate vocab */ }

When it happens

Trigger: Constructing SentencePieceUnigramModel from a tokenizer.json with model.byte_fallback == true where <0x00> or <0xFF> is absent, or where their ids differ by more than 0xFF.

Common situations: Tokenizers whose byte pieces were pruned during vocabulary filtering; models trained without byte fallback but with the flag turned on in exported config; conversion tools that reordered or dropped special pieces.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• The BOS, EOS, or UNK token is not present in the vocabulary.
• The vocabulary does not contain the required special token.
• ArgumentNullException
• vocab
• unkId must be a valid index in the vocabulary.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/0bbe9c536709468d. Report an issue: GitHub.