{"record":{"id":"6b6687c24357e3ff","repo":"dotnet/machinelearning","slug":"replace-normalizer-requires-a-string-or-regex-patt","errorCode":null,"errorMessage":"Replace normalizer requires a String or Regex pattern.","messagePattern":"Replace normalizer requires a String or Regex pattern\\.","errorType":"exception","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs","lineNumber":452,"sourceCode":"                if (pattern.TryGetProperty(\"Regex\", out JsonElement regex))\n                {\n                    if (regex.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"Replace normalizer 'Regex' pattern must be a string.\");\n                    }\n\n                    string regexPattern = regex.GetString()!;\n                    try\n                    {\n                        return new ReplaceStep(literal: null, new Regex(regexPattern, RegexOptions.CultureInvariant, _regexTimeout), content);\n                    }\n                    catch (ArgumentException ex)\n                    {\n                        throw new InvalidDataException($\"Replace normalizer has an invalid Regex pattern '{regexPattern}'.\", ex);\n                    }\n                }\n\n                throw new NotSupportedException(\"Replace normalizer requires a String or Regex pattern.\");\n            }\n\n            public override string Normalize(string text)\n            {\n                if (_regex is not null)\n                {\n                    // Hugging Face replaces the matched range with 'content' literally; escape '$' so Regex.Replace\n                    // does not interpret it as a substitution pattern (e.g. \"$0\", \"$&\") and diverge from the reference.\n                    string replacement = _content.IndexOf('$') < 0 ? _content : _content.Replace(\"$\", \"$$\");\n                    return _regex.Replace(text, replacement);\n                }\n\n                return string.IsNullOrEmpty(_literal) ? text : text.Replace(_literal, _content);\n            }\n        }\n\n        // Applies a SentencePiece precompiled charsmap by delegating to a charsmap-only SentencePieceNormalizer\n        // (no dummy prefix, no whitespace escaping, no whitespace stripping), reusing the existing DARTS engine.","sourceCodeStart":434,"sourceCodeEnd":470,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs#L434-L470","documentation":"A Replace normalizer's pattern object must contain either a 'String' member (literal replace) or a 'Regex' member (regex replace). If it contains neither, Create throws this NotSupportedException because the step cannot be constructed.","triggerScenarios":"tokenizer.json with {\"type\":\"Replace\",\"pattern\":{},\"content\":\"x\"} or a pattern object with misspelled/extra members (e.g. \"Literal\" or \"string\" lowercase).","commonSituations":"Hand-authored tokenizer.json; case-sensitive key typos; files converted between schema versions.","solutions":["Add exactly one of \"String\" or \"Regex\" inside the pattern object.","Check key casing: \"String\"/\"Regex\" with capital first letters.","Regenerate the normalizer entry from the HuggingFace tokenizers library to confirm the schema."],"exampleFix":"// before\n{\"type\": \"Replace\", \"pattern\": {}, \"content\": \"x\"}\n// after\n{\"type\": \"Replace\", \"pattern\": {\"String\": \"x\"}, \"content\": \"y\"}","handlingStrategy":"validation","validationCode":"var p = normalizer.GetProperty(\"pattern\");\nif (!p.TryGetProperty(\"String\", out _) && !p.TryGetProperty(\"Regex\", out _))\n    throw new Exception(\"Replace pattern must contain 'String' or 'Regex'\");","typeGuard":"bool HasValidPatternKind(JsonElement pattern) =>\n    (pattern.TryGetProperty(\"String\", out var s) && s.ValueKind == JsonValueKind.String) ||\n    (pattern.TryGetProperty(\"Regex\", out var r) && r.ValueKind == JsonValueKind.String);","tryCatchPattern":"try { LoadTokenizer(path); }\ncatch (NotSupportedException ex) when (ex.Message == \"Replace normalizer requires a String or Regex pattern.\") {\n    // regenerate the normalizer entry with a supported pattern kind\n}","preventionTips":["Check key casing exactly: String / Regex","Copy Replace entries from a working tokenizer.json as a template","Schema-validate tokenizer.json in CI"],"tags":["tokenizer","normalizer","missing-field","unsupported"],"backgroundTag":"missing-required-config-field","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}