{"record":{"id":"bec28358271281f0","repo":"dotnet/machinelearning","slug":"normalization-modelproto-normalizerspec-name-i","errorCode":null,"errorMessage":"Normalization '{modelProto.NormalizerSpec.Name}' is not supported.","messagePattern":"Normalization '(.+?)' is not supported\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/LlamaTokenizer.cs","lineNumber":54,"sourceCode":"            bool addBeginOfSentence = true,\n            bool addEndOfSentence = false,\n            IReadOnlyDictionary<string, int>? specialTokens = null)\n        {\n            ModelProto modelProto = ModelProto.Parser.ParseFrom(modelStream);\n\n            if (modelProto is null)\n            {\n                throw new ArgumentNullException(nameof(modelProto));\n            }\n\n            if (modelProto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Bpe)\n            {\n                throw new ArgumentException(\"The model type is not Bpe.\", nameof(modelProto));\n            }\n\n            if (modelProto.NormalizerSpec.Name != \"identity\" && !string.IsNullOrEmpty(modelProto.NormalizerSpec.Name))\n            {\n                throw new ArgumentException($\"Normalization '{modelProto.NormalizerSpec.Name}' is not supported.\", nameof(modelProto));\n            }\n\n            return new LlamaTokenizer(modelProto, addBeginOfSentence, addEndOfSentence, specialTokens);\n        }\n    }\n}\n","sourceCodeStart":36,"sourceCodeEnd":61,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/LlamaTokenizer.cs#L36-L61","documentation":"ArgumentException thrown during LlamaTokenizer.Create (after parsing the SentencePiece ModelProto) when the proto's normalizer spec names a normalizer this tokenizer does not implement. Create only supports BPE models with a known subset of normalizers; an unrecognized NormalizerSpec.Name leaves the tokenizer unable to reproduce training-time preprocessing, so creation fails instead of tokenizing incorrectly.","triggerScenarios":"Passing a ModelProto whose NormalizerSpec.Name is something like 'nmt_nfkc', 'nfkc', or 'precompiled_charsmap' to LlamaTokenizer.Create.","commonSituations":"Using a SentencePiece model trained with the default --normalization_rule_name=nmt_nfkc instead of identity; reusing models trained for other tokenizers.","solutions":["Retrain the model with --normalization_rule_name=identity","Pre-normalize text yourself and use a model with identity normalization","Check proto.NormalizerSpec.Name before calling Create and fall back to a tokenizer that supports it"],"exampleFix":"// before\nspm_train --input=corpus.txt --model_prefix=m --model_type=bpe  # default nmt_nfkc\n// after\nspm_train --input=corpus.txt --model_prefix=m --model_type=bpe --normalization_rule_name=identity","handlingStrategy":"validation","validationCode":"var name = proto.NormalizerSpec.Name; if (!string.IsNullOrEmpty(name) && name != \"identity\") throw new InvalidOperationException($\"Unsupported normalizer: {name}\");","typeGuard":null,"tryCatchPattern":"try { var tok = LlamaTokenizer.Create(proto); } catch (ArgumentException ex) { log.LogError(ex, \"Unsupported SentencePiece normalization\"); throw; }","preventionTips":["Train SentencePiece models with --normalization_rule_name=identity","Inspect NormalizerSpec.Name when onboarding new models","Route unsupported models to a tokenizer that handles their normalization"],"tags":["tokenizers","sentencepiece","normalization"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}