dotnet/machinelearning · error · InvalidDataException

Replace normalizer has an invalid Regex pattern '{regexPatte

Error message

Replace normalizer has an invalid Regex pattern '{regexPattern}'.

What it means

When the Replace normalizer's 'Regex' pattern string is syntactically invalid for the .NET Regex engine, new Regex throws ArgumentException, which is wrapped in this InvalidDataException including the offending pattern. The tokenizer.json cannot be loaded as-is.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs:448

                    return new ReplaceStep(literal.GetString() ?? "", regex: null, content);
                }

                if (pattern.TryGetProperty("Regex", out JsonElement regex))
                {
                    if (regex.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'Regex' pattern must be a string.");
                    }

                    string regexPattern = regex.GetString()!;
                    try
                    {
                        return new ReplaceStep(literal: null, new Regex(regexPattern, RegexOptions.CultureInvariant, _regexTimeout), content);
                    }
                    catch (ArgumentException ex)
                    {
                        throw new InvalidDataException($"Replace normalizer has an invalid Regex pattern '{regexPattern}'.", ex);
                    }
                }

                throw new NotSupportedException("Replace normalizer requires a String or Regex pattern.");
            }

            public override string Normalize(string text)
            {
                if (_regex is not null)
                {
                    // Hugging Face replaces the matched range with 'content' literally; escape '$' so Regex.Replace
                    // does not interpret it as a substitution pattern (e.g. "$0", "$&") and diverge from the reference.
                    string replacement = _content.IndexOf('$') < 0 ? _content : _content.Replace("$", "$$");
                    return _regex.Replace(text, replacement);
                }

                return string.IsNullOrEmpty(_literal) ? text : text.Replace(_literal, _content);
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Test the pattern with new Regex(pattern) in .NET and fix the reported syntax error.
2. Rewrite engine-specific constructs into .NET-compatible regex.
3. If a literal replace suffices, switch to a "String" pattern.
4. Regenerate tokenizer.json with a newer tokenizers version using .NET-compatible patterns.

Example fix

// before: unbalanced group
"pattern": {"Regex": "(\s"}
// after
"pattern": {"Regex": "(\s+)"}

static bool IsValidDotNetRegex(string pattern) {
    try { new Regex(pattern, RegexOptions.CultureInvariant, TimeSpan.FromSeconds(1)); return true; }
    catch (ArgumentException) { return false; }
}

bool IsDotNetCompatible(JsonElement pattern) => pattern.TryGetProperty("Regex", out var r) && r.ValueKind == JsonValueKind.String && IsValidDotNetRegex(r.GetString()!);

try { LoadTokenizer(path); }
catch (InvalidDataException ex) when (ex.Message.StartsWith("Replace normalizer has an invalid Regex pattern")) {
    // parse the pattern from the message, fix .NET-incompatible syntax, reload
}

When it happens

Trigger: tokenizer.json with a 'Regex' pattern using constructs unsupported or illegal in .NET (e.g. unmatched parenthesis, invalid group name, unsupported lookbehind of variable length, JS-specific syntax like (?<=>) or \p{Script=X} with invalid script).

Common situations: Regex authored for the Rust/Oniguruma tokenizers engine that .NET doesn't accept; escaping mistakes after JSON unescaping (\\s becoming wrong); very old tokenizer.json files.

Related errors

• Replace normalizer 'Regex' pattern must be a string.
• An 'added_tokens' entry must have a string 'content' and a n
• The tokenizer.json post_processor special token '{tokenName}
• The post-processor '{property}' token '{token}' with id {id}
• The tokenizer.json normalizer 'precompiled_charsmap' is not

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/3e19d2a9a8b45b12. Report an issue: GitHub.