dotnet/machinelearning · error · NotSupportedException

Unigram normalizer type

Error message

Unigram normalizer type '{type}' is not supported when loading a tokenizer.json with content-modifying normalizer steps.

What it means

SentencePieceNormalizationStep.Build supports a fixed set of normalizer types (Nmt, Precompiled, Prepend, etc.). Any unrecognized type string (or a missing 'type') throws NotSupportedException, because the library cannot apply a content-modifying normalizer it doesn't understand and silently mis-tokenizing would be worse.

Solutions

1. Upgrade Microsoft.ML.Tokenizers to a version supporting the normalizer type
2. Pre-normalize your text yourself and load the tokenizer without the normalizer, or edit tokenizer.json to a supported type (e.g. Precompiled/Prepend)
3. Use a different loader that supports the full Hugging Face normalizer set (e.g. Tokenizers.DotNet) for such models

Example fix

// before
"normalizer": { "type": "NFKC" }
// after
"normalizer": { "type": "Precompiled", "precompiled_charsmap": "..." }

static readonly HashSet<string> SupportedTypes = new() { "Precompiled", "Prepend", "Nmt", /* other types your library version supports */ };
if (normalizer.TryGetProperty("type", out var t) && t.ValueKind == JsonValueKind.String && !SupportedTypes.Contains(t.GetString()!))
    throw new NotSupportedException($"Normalizer type '{t.GetString()}' is not supported");

bool IsSupportedNormalizerType(JsonElement e) => e.TryGetProperty("type", out var t) && t.ValueKind == JsonValueKind.String && SupportedTypes.Contains(t.GetString()!);

try { LoadTokenizer(tokenizerJson); } catch (NotSupportedException ex) when (ex.Message.Contains("normalizer type")) { log.LogWarning(ex, "Unsupported normalizer; loading without it or pre-normalizing input"); }

When it happens

Trigger: Loading a tokenizer.json with a normalizer type not in the switch — e.g. "NFD", "NFKD", "Strip", "Replace", "Sequence", "Lowercase", or a typo like "precomiled".

Common situations: Tokenizer models (e.g. mBART, NLLB T5-style files) whose normalizers use Unicode normalization types this .NET loader doesn't implement; newer tokenizers versions emitting new normalizer types; misspelled type names in hand-authored files.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• A tokenizer.json normalizer entry must be a JSON object.
• ArgumentNullException
• Normalization ' ' is not supported.
• The Metaspace 'replacement
• The model type is not Bpe.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/cf60c6d4c1c762b6. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs:204

                    return new UnicodeStep(NormalizationForm.FormKD);

                case "Prepend":
                    {
                        if (normalizer.TryGetProperty("prepend", out JsonElement prependElement) &&
                            prependElement.ValueKind != JsonValueKind.String && prependElement.ValueKind != JsonValueKind.Null)
                        {
                            throw new InvalidDataException("The Prepend normalizer 'prepend' must be a string.");
                        }

                        string prepend = prependElement.ValueKind == JsonValueKind.String ? prependElement.GetString() ?? "" : "";
                        return new PrependStep(prepend);
                    }

                case "Nmt":
                    return NmtStep.Instance;

                default:
                    throw new NotSupportedException(
                        $"Unigram normalizer type '{type ?? "<missing>"}' is not supported when loading a tokenizer.json with content-modifying normalizer steps.");
            }
        }

        // Decodes a base64 'precompiled_charsmap' value, surfacing malformed input as InvalidDataException so callers
        // get a consistent, diagnostic failure for bad tokenizer.json files instead of a raw FormatException.
        internal static byte[] DecodePrecompiledCharsMap(string base64)
        {
            try
            {
                return Convert.FromBase64String(base64);
            }
            catch (FormatException ex)
            {
                throw new InvalidDataException("The tokenizer.json normalizer 'precompiled_charsmap' is not valid base64.", ex);
            }
        }

View on GitHub (pinned to 7b76e69cf9)