dotnet/machinelearning · error · InvalidDataException

Replace normalizer is missing its pattern.

Error message

Replace normalizer is missing its pattern.

What it means

The Replace normalizer step in tokenizer.json requires a 'pattern' property that must be a JSON object (with a 'String' or 'Regex' member). If the property is absent or is not an object, Create throws this InvalidDataException because there is nothing to replace.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs:421

            private readonly string _content;
            private readonly string? _literal;
            private readonly Regex? _regex;

            private ReplaceStep(string? literal, Regex? regex, string content)
            {
                _literal = literal;
                _regex = regex;
                _content = content;
            }

            public static ReplaceStep Create(JsonElement normalizer)
            {
                string content = normalizer.TryGetProperty("content", out JsonElement contentElement) && contentElement.ValueKind == JsonValueKind.String
                    ? contentElement.GetString() ?? "" : "";
                if (!normalizer.TryGetProperty("pattern", out JsonElement pattern) || pattern.ValueKind != JsonValueKind.Object)
                {
                    throw new InvalidDataException("Replace normalizer is missing its pattern.");
                }

                if (pattern.TryGetProperty("String", out JsonElement literal))
                {
                    if (literal.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'String' pattern must be a string.");
                    }

                    return new ReplaceStep(literal.GetString() ?? "", regex: null, content);
                }

                if (pattern.TryGetProperty("Regex", out JsonElement regex))
                {
                    if (regex.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'Regex' pattern must be a string.");
                    }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Add the missing pattern object: {"type": "Replace", "pattern": {"String": "old"}, "content": "new"}.
2. Check that 'pattern' is an object, not a bare string.
3. Regenerate tokenizer.json with the matching tokenizers library version.

Example fix

// before
{"type": "Replace", "content": " "}
// after
{"type": "Replace", "pattern": {"String": "\u3000"}, "content": " "}

var n = json.GetProperty("normalizer");
if (n.GetProperty("type").GetString() == "Replace" &&
    (!n.TryGetProperty("pattern", out var p) || p.ValueKind != JsonValueKind.Object))
    throw new Exception("tokenizer.json Replace normalizer needs a pattern object");

bool HasPatternObject(JsonElement n) => n.TryGetProperty("pattern", out var p) && p.ValueKind == JsonValueKind.Object;

try { LoadTokenizer(path); }
catch (InvalidDataException ex) when (ex.Message == "Replace normalizer is missing its pattern.") {
    // repair/replace tokenizer.json
}

When it happens

Trigger: Tokenizer.Create on a tokenizer.json whose normalizer entry of type Replace lacks a 'pattern' object, or has 'pattern' as a plain string/other JSON kind instead of an object.

Common situations: Hand-written tokenizer.json; converting from tokenizers library versions with a different Replace serialization; scripts that strip or rename JSON fields.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Replace normalizer 'String' pattern must be a string.
• Replace normalizer requires a String or Regex pattern.
• The content of the vocabulary file '{vocabFile}' is not vali
• Problems met when parsing JSON vocabulary object.{Environmen
• Problems met when parsing JSON vocabulary object.{Environmen

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5fa74bee5cca2c1a. Report an issue: GitHub.