dotnet/machinelearning · error · InvalidDataException

The pre_tokenizer 'replacement' must be a string.

Error message

The pre_tokenizer 'replacement' must be a string.

What it means

The Metaspace pre-tokenizer's 'replacement' property must be either a JSON string or null. It specifies the whitespace marker character used when replacing spaces; a non-string, non-null JSON value (number, boolean, object, array) cannot be read as a character, so SentencePieceTokenizer rejects the tokenizer.json with InvalidDataException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:1237

            string? type = GetStringOrNull(preTokenizer, "type");
            if (string.Equals(type, "Metaspace", StringComparison.OrdinalIgnoreCase))
            {
                if (preTokenizer.TryGetProperty("add_prefix_space", out JsonElement addPrefixElement))
                {
                    if (addPrefixElement.ValueKind != JsonValueKind.True && addPrefixElement.ValueKind != JsonValueKind.False)
                    {
                        throw new InvalidDataException("The pre_tokenizer 'add_prefix_space' must be a boolean.");
                    }

                    addDummyPrefix = addPrefixElement.GetBoolean();
                }

                if (preTokenizer.TryGetProperty("replacement", out JsonElement replacementElement))
                {
                    if (replacementElement.ValueKind != JsonValueKind.String && replacementElement.ValueKind != JsonValueKind.Null)
                    {
                        throw new InvalidDataException("The pre_tokenizer 'replacement' must be a string.");
                    }

                    // HF Metaspace's 'replacement' is the actual whitespace marker character. The SentencePiece model
                    // only supports U+2581 ('▁'); reject any other marker rather than silently not escaping spaces.
                    string? replacement = replacementElement.GetString();
                    if (replacement is not null && replacement != "\u2581") // U+2581 LOWER ONE EIGHTH BLOCK (▁)
                    {
                        throw new NotSupportedException(
                            $"The Metaspace 'replacement' '{replacement}' is not supported; only U+2581 ('\u2581') is supported.");
                    }

                    escapeWhiteSpaces = true;
                }

                if (preTokenizer.TryGetProperty("prepend_scheme", out JsonElement prependSchemeElement))
                {
                    string? scheme = prependSchemeElement.ValueKind == JsonValueKind.String ? prependSchemeElement.GetString() : null;
                    // "never" suppresses the dummy prefix; "always"/"first" keep the default (true)

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Edit tokenizer.json so pre_tokenizer.replacement is a JSON string, e.g. "replacement": "▁" (U+2581).
2. Use null (or omit the property) if no replacement marker is needed.
3. Regenerate tokenizer.json with the Hugging Face tokenizers save_pretrained flow so the Metaspace options keep correct types.
4. Lint/parse the file with Python tokenizers (Tokenizers.from_file) before loading in .NET to locate malformed fields.

Example fix

// before (tokenizer.json)
"pre_tokenizer": { "type": "Metaspace", "replacement": 9601 }
// after
"pre_tokenizer": { "type": "Metaspace", "replacement": "▁" }

// C# — check replacement type before loading
if (preTokenizer.TryGetProperty("replacement", out var r) &&
    r.ValueKind is not (JsonValueKind.String or JsonValueKind.Null))
{
    throw new InvalidDataException("pre_tokenizer.replacement must be a JSON string or null.");
}

static bool IsValidReplacement(JsonElement e) =>
    e.ValueKind == JsonValueKind.String || e.ValueKind == JsonValueKind.Null;

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) { /* inspect pre_tokenizer.replacement in tokenizer.json */ }

When it happens

Trigger: Loading a tokenizer.json where pre_tokenizer is Metaspace and pre_tokenizer.replacement is present with a JSON kind other than String or Null — e.g. "replacement": 9601 or true.

Common situations: Corrupted or hand-edited tokenizer.json files; conversion tools that wrote the replacement marker's codepoint as a number; copy/paste edits that dropped the quotes around "▁".

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• The pre_tokenizer 'add_prefix_space' must be a boolean.
• The Metaspace 'replacement' '{replacement}' is not supported
• The model type is not Bpe.
• Normalization '{modelProto.NormalizerSpec.Name}' is not supp
• ArgumentNullException

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/31ec03838eccaca0. Report an issue: GitHub.