{"record":{"id":"5fa74bee5cca2c1a","repo":"dotnet/machinelearning","slug":"replace-normalizer-is-missing-its-pattern","errorCode":null,"errorMessage":"Replace normalizer is missing its pattern.","messagePattern":"Replace normalizer is missing its pattern\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs","lineNumber":421,"sourceCode":"\n            private readonly string _content;\n            private readonly string? _literal;\n            private readonly Regex? _regex;\n\n            private ReplaceStep(string? literal, Regex? regex, string content)\n            {\n                _literal = literal;\n                _regex = regex;\n                _content = content;\n            }\n\n            public static ReplaceStep Create(JsonElement normalizer)\n            {\n                string content = normalizer.TryGetProperty(\"content\", out JsonElement contentElement) && contentElement.ValueKind == JsonValueKind.String\n                    ? contentElement.GetString() ?? \"\" : \"\";\n                if (!normalizer.TryGetProperty(\"pattern\", out JsonElement pattern) || pattern.ValueKind != JsonValueKind.Object)\n                {\n                    throw new InvalidDataException(\"Replace normalizer is missing its pattern.\");\n                }\n\n                if (pattern.TryGetProperty(\"String\", out JsonElement literal))\n                {\n                    if (literal.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"Replace normalizer 'String' pattern must be a string.\");\n                    }\n\n                    return new ReplaceStep(literal.GetString() ?? \"\", regex: null, content);\n                }\n\n                if (pattern.TryGetProperty(\"Regex\", out JsonElement regex))\n                {\n                    if (regex.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"Replace normalizer 'Regex' pattern must be a string.\");\n                    }","sourceCodeStart":403,"sourceCodeEnd":439,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs#L403-L439","documentation":"The Replace normalizer step in tokenizer.json requires a 'pattern' property that must be a JSON object (with a 'String' or 'Regex' member). If the property is absent or is not an object, Create throws this InvalidDataException because there is nothing to replace.","triggerScenarios":"Tokenizer.Create on a tokenizer.json whose normalizer entry of type Replace lacks a 'pattern' object, or has 'pattern' as a plain string/other JSON kind instead of an object.","commonSituations":"Hand-written tokenizer.json; converting from tokenizers library versions with a different Replace serialization; scripts that strip or rename JSON fields.","solutions":["Add the missing pattern object: {\"type\": \"Replace\", \"pattern\": {\"String\": \"old\"}, \"content\": \"new\"}.","Check that 'pattern' is an object, not a bare string.","Regenerate tokenizer.json with the matching tokenizers library version."],"exampleFix":"// before\n{\"type\": \"Replace\", \"content\": \" \"}\n// after\n{\"type\": \"Replace\", \"pattern\": {\"String\": \"\\u3000\"}, \"content\": \" \"}","handlingStrategy":"validation","validationCode":"var n = json.GetProperty(\"normalizer\");\nif (n.GetProperty(\"type\").GetString() == \"Replace\" &&\n    (!n.TryGetProperty(\"pattern\", out var p) || p.ValueKind != JsonValueKind.Object))\n    throw new Exception(\"tokenizer.json Replace normalizer needs a pattern object\");","typeGuard":"bool HasPatternObject(JsonElement n) => n.TryGetProperty(\"pattern\", out var p) && p.ValueKind == JsonValueKind.Object;","tryCatchPattern":"try { LoadTokenizer(path); }\ncatch (InvalidDataException ex) when (ex.Message == \"Replace normalizer is missing its pattern.\") {\n    // repair/replace tokenizer.json\n}","preventionTips":["Keep the exact HuggingFace Replace normalizer schema: {String|Regex} inside pattern object","Validate tokenizer.json against the tokenizers library before shipping","Avoid hand-written normalizer entries"],"tags":["tokenizer","normalizer","missing-field","json"],"backgroundTag":"missing-required-config-field","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}