{"record":{"id":"e6d06d3869ca529a","repo":"dotnet/machinelearning","slug":"replace-normalizer-regex-pattern-must-be-a-strin","errorCode":null,"errorMessage":"Replace normalizer 'Regex' pattern must be a string.","messagePattern":"Replace normalizer 'Regex' pattern must be a string\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs","lineNumber":438,"sourceCode":"                {\n                    throw new InvalidDataException(\"Replace normalizer is missing its pattern.\");\n                }\n\n                if (pattern.TryGetProperty(\"String\", out JsonElement literal))\n                {\n                    if (literal.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"Replace normalizer 'String' pattern must be a string.\");\n                    }\n\n                    return new ReplaceStep(literal.GetString() ?? \"\", regex: null, content);\n                }\n\n                if (pattern.TryGetProperty(\"Regex\", out JsonElement regex))\n                {\n                    if (regex.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"Replace normalizer 'Regex' pattern must be a string.\");\n                    }\n\n                    string regexPattern = regex.GetString()!;\n                    try\n                    {\n                        return new ReplaceStep(literal: null, new Regex(regexPattern, RegexOptions.CultureInvariant, _regexTimeout), content);\n                    }\n                    catch (ArgumentException ex)\n                    {\n                        throw new InvalidDataException($\"Replace normalizer has an invalid Regex pattern '{regexPattern}'.\", ex);\n                    }\n                }\n\n                throw new NotSupportedException(\"Replace normalizer requires a String or Regex pattern.\");\n            }\n\n            public override string Normalize(string text)\n            {","sourceCodeStart":420,"sourceCodeEnd":456,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs#L420-L456","documentation":"InvalidDataException thrown by SentencePieceNormalizationStep.Create while parsing the normalizer JSON of a tokenizer file: a \"Replace\" step declared its pattern with a \"Regex\" key, but the JSON value under that key is not a string (e.g. a number, object, or array). The deserializer expects a regular-expression pattern string, and the malformed node makes the normalization config unreadable.","triggerScenarios":"tokenizer.json contains {\"type\":\"Replace\",\"pattern\":{\"Regex\":{\"source\":\"...\"}},...} — a 'Regex' member present as a non-string (e.g. an object copied from another library's serialization).","commonSituations":"Copy-pasting pattern serialization from a JS/Rust tool where Regex is an object; programmatic generation emitting the regex as {source, flags}.","solutions":["Flatten the regex to a plain string: \"pattern\": {\"Regex\": \"\\\\s+\"}.","Move regex flags inline (e.g. (?i)...) since .NET Regex is constructed from the string.","Validate the file with the HuggingFace tokenizers library, which expects a string too."],"exampleFix":"// before\n\"pattern\": {\"Regex\": {\"source\": \"\\s+\", \"flags\": \"\"}}\n// after\n\"pattern\": {\"Regex\": \"\\s+\"}","handlingStrategy":"validation","validationCode":"if (pattern.TryGetProperty(\"Regex\", out var r) && r.ValueKind != JsonValueKind.String)\n    throw new Exception(\"Replace 'Regex' pattern must be a JSON string\");","typeGuard":"bool IsRegexPattern(JsonElement pattern) => pattern.TryGetProperty(\"Regex\", out var r) && r.ValueKind == JsonValueKind.String;","tryCatchPattern":"try { LoadTokenizer(path); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"'Regex' pattern must be a string\")) {\n    // flatten {source, flags} objects into a single pattern string\n}","preventionTips":["Emit Regex patterns as plain strings, never {source, flags} objects","Embed flags inline (e.g. (?i)) rather than as a separate field","Round-trip tokenizer.json through the HuggingFace tokenizers lib to normalize the schema"],"tags":["tokenizer","normalizer","type-mismatch","regex"],"backgroundTag":"type-mismatch","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}