dotnet/machinelearning · error · NotSupportedException

The Metaspace 'replacement' '{replacement}' is not supported

Error message

The Metaspace 'replacement' '{replacement}' is not supported; only U+2581 ('▁') is supported.

What it means

The SentencePiece model format only supports U+2581 ('▁', lower one eighth block) as the Metaspace whitespace marker. If tokenizer.json's Metaspace pre_tokenizer specifies a different 'replacement' character, Microsoft.ML.Tokenizers throws NotSupportedException instead of silently failing to escape spaces during encoding.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:1245

                        throw new InvalidDataException("The pre_tokenizer 'add_prefix_space' must be a boolean.");
                    }

                    addDummyPrefix = addPrefixElement.GetBoolean();
                }

                if (preTokenizer.TryGetProperty("replacement", out JsonElement replacementElement))
                {
                    if (replacementElement.ValueKind != JsonValueKind.String && replacementElement.ValueKind != JsonValueKind.Null)
                    {
                        throw new InvalidDataException("The pre_tokenizer 'replacement' must be a string.");
                    }

                    // HF Metaspace's 'replacement' is the actual whitespace marker character. The SentencePiece model
                    // only supports U+2581 ('▁'); reject any other marker rather than silently not escaping spaces.
                    string? replacement = replacementElement.GetString();
                    if (replacement is not null && replacement != "\u2581") // U+2581 LOWER ONE EIGHTH BLOCK (▁)
                    {
                        throw new NotSupportedException(
                            $"The Metaspace 'replacement' '{replacement}' is not supported; only U+2581 ('\u2581') is supported.");
                    }

                    escapeWhiteSpaces = true;
                }

                if (preTokenizer.TryGetProperty("prepend_scheme", out JsonElement prependSchemeElement))
                {
                    string? scheme = prependSchemeElement.ValueKind == JsonValueKind.String ? prependSchemeElement.GetString() : null;
                    // "never" suppresses the dummy prefix; "always"/"first" keep the default (true)
                    if (string.Equals(scheme, "never", StringComparison.OrdinalIgnoreCase))
                    {
                        addDummyPrefix = false;
                    }
                }
            }
            else if (string.Equals(type, "Sequence", StringComparison.OrdinalIgnoreCase) &&
                     preTokenizer.TryGetProperty("pretokenizers", out JsonElement preTokenizersElement) &&

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Set pre_tokenizer.replacement to "▁" (U+2581) in tokenizer.json.
2. Set replacement to null or remove it to fall back to the model's default behavior.
3. Verify the tokenizer actually belongs to a SentencePiece model; if it uses a different space marker, load it with the appropriate tokenizer type (e.g. BPE) rather than SentencePieceTokenizer.
4. Regenerate the tokenizer artifact from the original SentencePiece training run so the standard ▁ marker is produced.

Example fix

// before (tokenizer.json)
"pre_tokenizer": { "type": "Metaspace", "replacement": "_" }
// after
"pre_tokenizer": { "type": "Metaspace", "replacement": "▁" }

// C# — verify replacement marker before loading
if (preTokenizer.TryGetProperty("replacement", out var r) &&
    r.ValueKind == JsonValueKind.String &&
    r.GetString() is string s && s != "\u2581")
{
    throw new NotSupportedException($"Metaspace replacement '{s}' unsupported; use U+2581.");
}

static bool IsSupportedReplacement(string? r) => r is null || r == "\u2581";

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (NotSupportedException ex) { /* the model is not SentencePiece-compatible; use correct tokenizer type */ }

When it happens

Trigger: Loading a tokenizer.json whose Metaspace pre_tokenizer.replacement is a non-null string other than "▁" — e.g. "replacement": "Ġ" (as in GPT-style byte-level BPE Metaspace variants) or "_".

Common situations: Tokenizers exported from byte-level pipelines that use different space markers; mixing pre_tokenizer configs across model families; hand-porting a BPE tokenizer.json to be loaded as SentencePiece.

Related errors

• The pre_tokenizer 'add_prefix_space' must be a boolean.
• The pre_tokenizer 'replacement' must be a string.
• The model type is not Bpe.
• Normalization '{modelProto.NormalizerSpec.Name}' is not supp
• ArgumentNullException

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e9de060c133e04a2. Report an issue: GitHub.