dotnet/machinelearning · error · InvalidDataException

Replace normalizer 'String' pattern must be a string.

Error message

Replace normalizer 'String' pattern must be a string.

What it means

If a Replace normalizer's pattern object has a 'String' member, its value must be a JSON string — the literal substring to replace. Any other JSON kind (number, null, object) triggers this InvalidDataException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs:428

                _literal = literal;
                _regex = regex;
                _content = content;
            }

            public static ReplaceStep Create(JsonElement normalizer)
            {
                string content = normalizer.TryGetProperty("content", out JsonElement contentElement) && contentElement.ValueKind == JsonValueKind.String
                    ? contentElement.GetString() ?? "" : "";
                if (!normalizer.TryGetProperty("pattern", out JsonElement pattern) || pattern.ValueKind != JsonValueKind.Object)
                {
                    throw new InvalidDataException("Replace normalizer is missing its pattern.");
                }

                if (pattern.TryGetProperty("String", out JsonElement literal))
                {
                    if (literal.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'String' pattern must be a string.");
                    }

                    return new ReplaceStep(literal.GetString() ?? "", regex: null, content);
                }

                if (pattern.TryGetProperty("Regex", out JsonElement regex))
                {
                    if (regex.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("Replace normalizer 'Regex' pattern must be a string.");
                    }

                    string regexPattern = regex.GetString()!;
                    try
                    {
                        return new ReplaceStep(literal: null, new Regex(regexPattern, RegexOptions.CultureInvariant, _regexTimeout), content);
                    }
                    catch (ArgumentException ex)

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Quote the value: "pattern": {"String": "literal text"}.
2. If you intended a regex, use the "Regex" member instead of "String".
3. Validate tokenizer.json with the Python tokenizers library before using it in .NET.

Example fix

// before
"pattern": {"String": 42}
// after
"pattern": {"String": "42"}

if (pattern.TryGetProperty("String", out var s) && s.ValueKind != JsonValueKind.String)
    throw new Exception("Replace 'String' pattern must be a JSON string");

bool IsStringPattern(JsonElement pattern) => pattern.TryGetProperty("String", out var s) && s.ValueKind == JsonValueKind.String;

try { LoadTokenizer(path); }
catch (InvalidDataException ex) when (ex.Message.Contains("'String' pattern must be a string")) {
    // fix the JSON member type and retry
}

When it happens

Trigger: tokenizer.json contains {"type":"Replace","pattern":{"String":123},...} or {"String":null} — a 'String' member present with a non-string value.

Common situations: Generating tokenizer.json from code that didn't quote the value; YAML-to-JSON conversion dropping quotes; hand edits.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• A post_processor template 'SpecialToken.id' must be a string
• The tokenizer.json post_processor special token '{tokenName}
• Replace normalizer is missing its pattern.
• Replace normalizer 'Regex' pattern must be a string.
• unknown option type

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/421b2197e098113a. Report an issue: GitHub.