dotnet/machinelearning · error · InvalidDataException

Replace normalizer 'Regex' pattern must be a string.

Error message

Replace normalizer 'Regex' pattern must be a string.

What it means

InvalidDataException thrown by SentencePieceNormalizationStep.Create while parsing the normalizer JSON of a tokenizer file: a "Replace" step declared its pattern with a "Regex" key, but the JSON value under that key is not a string (e.g. a number, object, or array). The deserializer expects a regular-expression pattern string, and the malformed node makes the normalization config unreadable.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs:438

                {
                    throw new InvalidDataException("Replace normalizer is missing its pattern.");
                }

                if (pattern.TryGetProperty("String", out JsonElement literal))
                {
                    if (literal.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'String' pattern must be a string.");
                    }

                    return new ReplaceStep(literal.GetString() ?? "", regex: null, content);
                }

                if (pattern.TryGetProperty("Regex", out JsonElement regex))
                {
                    if (regex.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'Regex' pattern must be a string.");
                    }

                    string regexPattern = regex.GetString()!;
                    try
                    {
                        return new ReplaceStep(literal: null, new Regex(regexPattern, RegexOptions.CultureInvariant, _regexTimeout), content);
                    }
                    catch (ArgumentException ex)
                    {
                        throw new InvalidDataException($"Replace normalizer has an invalid Regex pattern '{regexPattern}'.", ex);
                    }
                }

                throw new NotSupportedException("Replace normalizer requires a String or Regex pattern.");
            }

            public override string Normalize(string text)
            {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Flatten the regex to a plain string: "pattern": {"Regex": "\\s+"}.
2. Move regex flags inline (e.g. (?i)...) since .NET Regex is constructed from the string.
3. Validate the file with the HuggingFace tokenizers library, which expects a string too.

Example fix

// before
"pattern": {"Regex": {"source": "\s+", "flags": ""}}
// after
"pattern": {"Regex": "\s+"}

if (pattern.TryGetProperty("Regex", out var r) && r.ValueKind != JsonValueKind.String)
    throw new Exception("Replace 'Regex' pattern must be a JSON string");

bool IsRegexPattern(JsonElement pattern) => pattern.TryGetProperty("Regex", out var r) && r.ValueKind == JsonValueKind.String;

try { LoadTokenizer(path); }
catch (InvalidDataException ex) when (ex.Message.Contains("'Regex' pattern must be a string")) {
    // flatten {source, flags} objects into a single pattern string
}

When it happens

Trigger: tokenizer.json contains {"type":"Replace","pattern":{"Regex":{"source":"..."}},...} — a 'Regex' member present as a non-string (e.g. an object copied from another library's serialization).

Common situations: Copy-pasting pattern serialization from a JS/Rust tool where Regex is an object; programmatic generation emitting the regex as {source, flags}.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• Replace normalizer 'String' pattern must be a string.
• Replace normalizer has an invalid Regex pattern '{regexPatte
• The tokenizer.json model 'unk_id' property must be a number
• Each entry in 'model.vocab' must be a [string piece, number
• A post_processor template 'SpecialToken.id' must be a string

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e6d06d3869ca529a. Report an issue: GitHub.