dotnet/machinelearning · error · NotSupportedException

The tokenizer.json model has a null 'unk_id' but does not…

Error message

The tokenizer.json model has a null 'unk_id' but does not enable 'byte_fallback'; a Unigram model without an unknown token is only supported when byte_fallback is enabled.

What it means

When unk_id is null the model has no unknown token, so out-of-vocabulary text can only be encoded via byte fallback. If byte_fallback is not enabled, no valid encoding exists for OOV input, so the loader throws NotSupportedException up front rather than producing invalid token ids at encode time.

Solutions

Set byte_fallback: true in the model section (and include the <0x00>-<0xFF> byte pieces in the vocab, as the original training did).
Alternatively give the model a real unknown token: set unk_id to the index of an '<unk>' piece present in vocab.
Retrain/re-export the tokenizer with unk_id defined if byte fallback is not desired.

Example fix

// before
"model": { "type": "Unigram", "unk_id": null, "byte_fallback": false, "vocab": [...] }
// after
"model": { "type": "Unigram", "unk_id": null, "byte_fallback": true, "vocab": [["<0x00>", ...], ..., ["<0xFF>", ...], ...] }

Defensive patterns

Strategy: validation

Validate before calling

var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty("model");
bool unkNull = model.GetProperty("unk_id").ValueKind == JsonValueKind.Null;
bool bf = model.TryGetProperty("byte_fallback", out var b) && b.ValueKind == JsonValueKind.True;
if (unkNull && !bf)
    throw new InvalidOperationException("null unk_id requires byte_fallback: true (with byte pieces in vocab).");

Try / catch

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (NotSupportedException ex) when (ex.Message.Contains("byte_fallback")) { /* choose a tokenizer with an unk token or byte fallback */ }

Prevention

Only use unk_id: null with models trained with byte fallback enabled.
Keep the <0x00>..<0xFF> byte pieces in vocab when byte_fallback is true.
Match training-time SentencePiece flags when exporting tokenizer.json.

When it happens

Trigger: CreateFromTokenizerJson where model.unk_id is null and model.byte_fallback is absent or false.

Common situations: Custom-trained SentencePiece models trained with --hard_vocab_limit and no unk piece and without byte fallback; hand-edited tokenizer.json setting unk_id to null; stripping <unk> from the vocab without enabling byte_fallback.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/27f98d9f7a1ebea4. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:656

                }

                string? piece = entry[0].GetString();
                if (piece is null)
                {
                    throw new InvalidDataException("A piece string in 'model.vocab' is null.");
                }

                vocab.Add((piece, entry[1].GetSingle()));
            }

            if (unkIsNull)
            {
                // Without an unknown token the only way to represent out-of-vocabulary input is byte fallback; a model
                // with neither cannot encode OOV text, so reject that combination up front rather than emitting an
                // invalid token id at encode time.
                if (!byteFallback)
                {
                    throw new NotSupportedException("The tokenizer.json model has a null 'unk_id' but does not enable 'byte_fallback'; a Unigram model without an unknown token is only supported when byte_fallback is enabled.");
                }
            }
            else if (unkId < 0 || unkId >= vocab.Count)
            {
                throw new InvalidDataException($"The tokenizer.json model 'unk_id' ({unkId}) is out of range for a vocabulary of {vocab.Count} pieces.");
            }

            // Extract normalizer settings
            byte[]? precompiledCharsMap = null;
            bool addDummyPrefix = true;
            // HF tokenizer.json has no remove_extra_whitespaces flag; SpmConverter encodes that behavior as
            // explicit normalizer steps (a right-Strip plus a Replace collapsing runs of spaces). Deduce it from
            // those steps, defaulting to false when absent to match the HF fast-tokenizer runtime.
            bool removeExtraWhitespaces = false;
            // When the normalizer has content-modifying steps that the charsmap + removeExtraWhitespaces
            // approximation cannot represent (per-character Replace, Lowercase, Unicode normalization, ...),
            // apply the full normalizer chain (charsmap included) before the Metaspace pass instead.
            SentencePieceNormalizationStep? chainNormalizer = null;

View on GitHub (pinned to 7b76e69cf9)