dotnet/machinelearning · error · NotSupportedException

Replace normalizer requires a String or Regex pattern.

Error message

Replace normalizer requires a String or Regex pattern.

What it means

A Replace normalizer's pattern object must contain either a 'String' member (literal replace) or a 'Regex' member (regex replace). If it contains neither, Create throws this NotSupportedException because the step cannot be constructed.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs:452

                if (pattern.TryGetProperty("Regex", out JsonElement regex))
                {
                    if (regex.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'Regex' pattern must be a string.");
                    }

                    string regexPattern = regex.GetString()!;
                    try
                    {
                        return new ReplaceStep(literal: null, new Regex(regexPattern, RegexOptions.CultureInvariant, _regexTimeout), content);
                    }
                    catch (ArgumentException ex)
                    {
                        throw new InvalidDataException($"Replace normalizer has an invalid Regex pattern '{regexPattern}'.", ex);
                    }
                }

                throw new NotSupportedException("Replace normalizer requires a String or Regex pattern.");
            }

            public override string Normalize(string text)
            {
                if (_regex is not null)
                {
                    // Hugging Face replaces the matched range with 'content' literally; escape '$' so Regex.Replace
                    // does not interpret it as a substitution pattern (e.g. "$0", "$&") and diverge from the reference.
                    string replacement = _content.IndexOf('$') < 0 ? _content : _content.Replace("$", "$$");
                    return _regex.Replace(text, replacement);
                }

                return string.IsNullOrEmpty(_literal) ? text : text.Replace(_literal, _content);
            }
        }

        // Applies a SentencePiece precompiled charsmap by delegating to a charsmap-only SentencePieceNormalizer
        // (no dummy prefix, no whitespace escaping, no whitespace stripping), reusing the existing DARTS engine.

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Add exactly one of "String" or "Regex" inside the pattern object.
2. Check key casing: "String"/"Regex" with capital first letters.
3. Regenerate the normalizer entry from the HuggingFace tokenizers library to confirm the schema.

Example fix

// before
{"type": "Replace", "pattern": {}, "content": "x"}
// after
{"type": "Replace", "pattern": {"String": "x"}, "content": "y"}

var p = normalizer.GetProperty("pattern");
if (!p.TryGetProperty("String", out _) && !p.TryGetProperty("Regex", out _))
    throw new Exception("Replace pattern must contain 'String' or 'Regex'");

bool HasValidPatternKind(JsonElement pattern) =>
    (pattern.TryGetProperty("String", out var s) && s.ValueKind == JsonValueKind.String) ||
    (pattern.TryGetProperty("Regex", out var r) && r.ValueKind == JsonValueKind.String);

try { LoadTokenizer(path); }
catch (NotSupportedException ex) when (ex.Message == "Replace normalizer requires a String or Regex pattern.") {
    // regenerate the normalizer entry with a supported pattern kind
}

When it happens

Trigger: tokenizer.json with {"type":"Replace","pattern":{},"content":"x"} or a pattern object with misspelled/extra members (e.g. "Literal" or "string" lowercase).

Common situations: Hand-authored tokenizer.json; case-sensitive key typos; files converted between schema versions.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Replace normalizer is missing its pattern.
• The tokenizer.json model does not contain an 'unk_id' proper
• Replace normalizer 'String' pattern must be a string.
• Replace normalizer 'Regex' pattern must be a string.
• Replace normalizer has an invalid Regex pattern '{regexPatte

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/6b6687c24357e3ff. Report an issue: GitHub.