{"record":{"id":"cf60c6d4c1c762b6","repo":"dotnet/machinelearning","slug":"unigram-normalizer-type-type-is-not-supported","errorCode":null,"errorMessage":"Unigram normalizer type '{type}' is not supported when loading a tokenizer.json with content-modifying normalizer steps.","messagePattern":"Unigram normalizer type '(.+?)' is not supported when loading a tokenizer\\.json with content-modifying normalizer steps\\.","errorType":"exception","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs","lineNumber":204,"sourceCode":"                    return new UnicodeStep(NormalizationForm.FormKD);\n\n                case \"Prepend\":\n                    {\n                        if (normalizer.TryGetProperty(\"prepend\", out JsonElement prependElement) &&\n                            prependElement.ValueKind != JsonValueKind.String && prependElement.ValueKind != JsonValueKind.Null)\n                        {\n                            throw new InvalidDataException(\"The Prepend normalizer 'prepend' must be a string.\");\n                        }\n\n                        string prepend = prependElement.ValueKind == JsonValueKind.String ? prependElement.GetString() ?? \"\" : \"\";\n                        return new PrependStep(prepend);\n                    }\n\n                case \"Nmt\":\n                    return NmtStep.Instance;\n\n                default:\n                    throw new NotSupportedException(\n                        $\"Unigram normalizer type '{type ?? \"<missing>\"}' is not supported when loading a tokenizer.json with content-modifying normalizer steps.\");\n            }\n        }\n\n        // Decodes a base64 'precompiled_charsmap' value, surfacing malformed input as InvalidDataException so callers\n        // get a consistent, diagnostic failure for bad tokenizer.json files instead of a raw FormatException.\n        internal static byte[] DecodePrecompiledCharsMap(string base64)\n        {\n            try\n            {\n                return Convert.FromBase64String(base64);\n            }\n            catch (FormatException ex)\n            {\n                throw new InvalidDataException(\"The tokenizer.json normalizer 'precompiled_charsmap' is not valid base64.\", ex);\n            }\n        }\n","sourceCodeStart":186,"sourceCodeEnd":222,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs#L186-L222","documentation":"SentencePieceNormalizationStep.Build supports a fixed set of normalizer types (Nmt, Precompiled, Prepend, etc.). Any unrecognized type string (or a missing 'type') throws NotSupportedException, because the library cannot apply a content-modifying normalizer it doesn't understand and silently mis-tokenizing would be worse.","triggerScenarios":"Loading a tokenizer.json with a normalizer type not in the switch — e.g. \"NFD\", \"NFKD\", \"Strip\", \"Replace\", \"Sequence\", \"Lowercase\", or a typo like \"precomiled\".","commonSituations":"Tokenizer models (e.g. mBART, NLLB T5-style files) whose normalizers use Unicode normalization types this .NET loader doesn't implement; newer tokenizers versions emitting new normalizer types; misspelled type names in hand-authored files.","solutions":["Upgrade Microsoft.ML.Tokenizers to a version supporting the normalizer type","Pre-normalize your text yourself and load the tokenizer without the normalizer, or edit tokenizer.json to a supported type (e.g. Precompiled/Prepend)","Use a different loader that supports the full Hugging Face normalizer set (e.g. Tokenizers.DotNet) for such models"],"exampleFix":"// before\n\"normalizer\": { \"type\": \"NFKC\" }\n// after\n\"normalizer\": { \"type\": \"Precompiled\", \"precompiled_charsmap\": \"...\" }","handlingStrategy":"try-catch","validationCode":"static readonly HashSet<string> SupportedTypes = new() { \"Precompiled\", \"Prepend\", \"Nmt\", /* other types your library version supports */ };\nif (normalizer.TryGetProperty(\"type\", out var t) && t.ValueKind == JsonValueKind.String && !SupportedTypes.Contains(t.GetString()!))\n    throw new NotSupportedException($\"Normalizer type '{t.GetString()}' is not supported\");","typeGuard":"bool IsSupportedNormalizerType(JsonElement e) => e.TryGetProperty(\"type\", out var t) && t.ValueKind == JsonValueKind.String && SupportedTypes.Contains(t.GetString()!);","tryCatchPattern":"try { LoadTokenizer(tokenizerJson); } catch (NotSupportedException ex) when (ex.Message.Contains(\"normalizer type\")) { log.LogWarning(ex, \"Unsupported normalizer; loading without it or pre-normalizing input\"); }","preventionTips":["Check which normalizer types your Microsoft.ML.Tokenizers version supports before choosing models","Pin library versions and review release notes for new normalizer support","Pre-normalize exotic models' text yourself and strip the normalizer from tokenizer.json"],"tags":["tokenizers","unsupported","sentencepiece"],"backgroundTag":"unsupported-operation","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}