{"record":{"id":"3e19d2a9a8b45b12","repo":"dotnet/machinelearning","slug":"replace-normalizer-has-an-invalid-regex-pattern","errorCode":null,"errorMessage":"Replace normalizer has an invalid Regex pattern '{regexPattern}'.","messagePattern":"Replace normalizer has an invalid Regex pattern '(.+?)'\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs","lineNumber":448,"sourceCode":"\n                    return new ReplaceStep(literal.GetString() ?? \"\", regex: null, content);\n                }\n\n                if (pattern.TryGetProperty(\"Regex\", out JsonElement regex))\n                {\n                    if (regex.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"Replace normalizer 'Regex' pattern must be a string.\");\n                    }\n\n                    string regexPattern = regex.GetString()!;\n                    try\n                    {\n                        return new ReplaceStep(literal: null, new Regex(regexPattern, RegexOptions.CultureInvariant, _regexTimeout), content);\n                    }\n                    catch (ArgumentException ex)\n                    {\n                        throw new InvalidDataException($\"Replace normalizer has an invalid Regex pattern '{regexPattern}'.\", ex);\n                    }\n                }\n\n                throw new NotSupportedException(\"Replace normalizer requires a String or Regex pattern.\");\n            }\n\n            public override string Normalize(string text)\n            {\n                if (_regex is not null)\n                {\n                    // Hugging Face replaces the matched range with 'content' literally; escape '$' so Regex.Replace\n                    // does not interpret it as a substitution pattern (e.g. \"$0\", \"$&\") and diverge from the reference.\n                    string replacement = _content.IndexOf('$') < 0 ? _content : _content.Replace(\"$\", \"$$\");\n                    return _regex.Replace(text, replacement);\n                }\n\n                return string.IsNullOrEmpty(_literal) ? text : text.Replace(_literal, _content);\n            }","sourceCodeStart":430,"sourceCodeEnd":466,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs#L430-L466","documentation":"When the Replace normalizer's 'Regex' pattern string is syntactically invalid for the .NET Regex engine, new Regex throws ArgumentException, which is wrapped in this InvalidDataException including the offending pattern. The tokenizer.json cannot be loaded as-is.","triggerScenarios":"tokenizer.json with a 'Regex' pattern using constructs unsupported or illegal in .NET (e.g. unmatched parenthesis, invalid group name, unsupported lookbehind of variable length, JS-specific syntax like (?<=>) or \\p{Script=X} with invalid script).","commonSituations":"Regex authored for the Rust/Oniguruma tokenizers engine that .NET doesn't accept; escaping mistakes after JSON unescaping (\\\\s becoming wrong); very old tokenizer.json files.","solutions":["Test the pattern with new Regex(pattern) in .NET and fix the reported syntax error.","Rewrite engine-specific constructs into .NET-compatible regex.","If a literal replace suffices, switch to a \"String\" pattern.","Regenerate tokenizer.json with a newer tokenizers version using .NET-compatible patterns."],"exampleFix":"// before: unbalanced group\n\"pattern\": {\"Regex\": \"(\\s\"}\n// after\n\"pattern\": {\"Regex\": \"(\\s+)\"}","handlingStrategy":"try-catch","validationCode":"static bool IsValidDotNetRegex(string pattern) {\n    try { new Regex(pattern, RegexOptions.CultureInvariant, TimeSpan.FromSeconds(1)); return true; }\n    catch (ArgumentException) { return false; }\n}","typeGuard":"bool IsDotNetCompatible(JsonElement pattern) => pattern.TryGetProperty(\"Regex\", out var r) && r.ValueKind == JsonValueKind.String && IsValidDotNetRegex(r.GetString()!);","tryCatchPattern":"try { LoadTokenizer(path); }\ncatch (InvalidDataException ex) when (ex.Message.StartsWith(\"Replace normalizer has an invalid Regex pattern\")) {\n    // parse the pattern from the message, fix .NET-incompatible syntax, reload\n}","preventionTips":["Test every custom regex with .NET Regex before embedding it in tokenizer.json","Avoid Oniguruma/PCRE-only syntax","Prefer String patterns when literal replacement suffices"],"tags":["tokenizer","regex","normalizer","invalid-data"],"backgroundTag":"invalid-regex-pattern","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}