dotnet/machinelearning · error · ArgumentException

Normalization '{modelProto.NormalizerSpec.Name}' is not supp

Error message

Normalization '{modelProto.NormalizerSpec.Name}' is not supported.

What it means

ArgumentException thrown during LlamaTokenizer.Create (after parsing the SentencePiece ModelProto) when the proto's normalizer spec names a normalizer this tokenizer does not implement. Create only supports BPE models with a known subset of normalizers; an unrecognized NormalizerSpec.Name leaves the tokenizer unable to reproduce training-time preprocessing, so creation fails instead of tokenizing incorrectly.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/LlamaTokenizer.cs:54

            bool addBeginOfSentence = true,
            bool addEndOfSentence = false,
            IReadOnlyDictionary<string, int>? specialTokens = null)
        {
            ModelProto modelProto = ModelProto.Parser.ParseFrom(modelStream);

            if (modelProto is null)
            {
                throw new ArgumentNullException(nameof(modelProto));
            }

            if (modelProto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Bpe)
            {
                throw new ArgumentException("The model type is not Bpe.", nameof(modelProto));
            }

            if (modelProto.NormalizerSpec.Name != "identity" && !string.IsNullOrEmpty(modelProto.NormalizerSpec.Name))
            {
                throw new ArgumentException($"Normalization '{modelProto.NormalizerSpec.Name}' is not supported.", nameof(modelProto));
            }

            return new LlamaTokenizer(modelProto, addBeginOfSentence, addEndOfSentence, specialTokens);
        }
    }
}

View on GitHub (pinned to 7b76e69cf9)

Solutions

Retrain the model with --normalization_rule_name=identity
Pre-normalize text yourself and use a model with identity normalization
Check proto.NormalizerSpec.Name before calling Create and fall back to a tokenizer that supports it

Example fix

// before
spm_train --input=corpus.txt --model_prefix=m --model_type=bpe  # default nmt_nfkc
// after
spm_train --input=corpus.txt --model_prefix=m --model_type=bpe --normalization_rule_name=identity

Defensive patterns

Strategy: validation

Validate before calling

var name = proto.NormalizerSpec.Name; if (!string.IsNullOrEmpty(name) && name != "identity") throw new InvalidOperationException($"Unsupported normalizer: {name}");

Try / catch

try { var tok = LlamaTokenizer.Create(proto); } catch (ArgumentException ex) { log.LogError(ex, "Unsupported SentencePiece normalization"); throw; }

Prevention

Train SentencePiece models with --normalization_rule_name=identity
Inspect NormalizerSpec.Name when onboarding new models
Route unsupported models to a tokenizer that handles their normalization

When it happens

Trigger: Passing a ModelProto whose NormalizerSpec.Name is something like 'nmt_nfkc', 'nfkc', or 'precompiled_charsmap' to LlamaTokenizer.Create.

Common situations: Using a SentencePiece model trained with the default --normalization_rule_name=nmt_nfkc instead of identity; reusing models trained for other tokenizers.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/bec28358271281f0. Report an issue: GitHub.