dotnet/machinelearning · error · InvalidDataException

The pre_tokenizer 'add_prefix_space' must be a boolean.

Error message

The pre_tokenizer 'add_prefix_space' must be a boolean.

What it means

When building a SentencePieceTokenizer from a tokenizer.json, the Metaspace pre-tokenizer's 'add_prefix_space' option must be JSON true or false. The library reads this property to decide whether to prepend the ▁ dummy prefix to input text; any other JSON kind (string, number, null, object) cannot be interpreted as a boolean, so loading fails with InvalidDataException rather than guessing a default.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:1227

            throw new NotSupportedException(
                $"The tokenizer.json pre_tokenizer type '{type ?? "<missing>"}' is not supported; only Metaspace, WhitespaceSplit, Whitespace, and Sequence are handled.");
        }

        private static void ExtractMetaspaceSettings(JsonElement preTokenizer, ref bool addDummyPrefix, ref bool escapeWhiteSpaces)
        {
            if (preTokenizer.ValueKind != JsonValueKind.Object)
            {
                return;
            }

            string? type = GetStringOrNull(preTokenizer, "type");
            if (string.Equals(type, "Metaspace", StringComparison.OrdinalIgnoreCase))
            {
                if (preTokenizer.TryGetProperty("add_prefix_space", out JsonElement addPrefixElement))
                {
                    if (addPrefixElement.ValueKind != JsonValueKind.True && addPrefixElement.ValueKind != JsonValueKind.False)
                    {
                        throw new InvalidDataException("The pre_tokenizer 'add_prefix_space' must be a boolean.");
                    }

                    addDummyPrefix = addPrefixElement.GetBoolean();
                }

                if (preTokenizer.TryGetProperty("replacement", out JsonElement replacementElement))
                {
                    if (replacementElement.ValueKind != JsonValueKind.String && replacementElement.ValueKind != JsonValueKind.Null)
                    {
                        throw new InvalidDataException("The pre_tokenizer 'replacement' must be a string.");
                    }

                    // HF Metaspace's 'replacement' is the actual whitespace marker character. The SentencePiece model
                    // only supports U+2581 ('▁'); reject any other marker rather than silently not escaping spaces.
                    string? replacement = replacementElement.GetString();
                    if (replacement is not null && replacement != "\u2581") // U+2581 LOWER ONE EIGHTH BLOCK (▁)
                    {
                        throw new NotSupportedException(

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Open tokenizer.json and change pre_tokenizer.add_prefix_space to a literal JSON boolean: true or false.
2. Remove the add_prefix_space property entirely if you want the library default behavior (it is only validated when present).
3. Regenerate tokenizer.json with the Hugging Face tokenizers library (save_pretrained) instead of hand-editing, ensuring the Metaspace pre-tokenizer options keep their native types.
4. Validate the tokenizer.json with the HF tokenizers Python library (Tokenizers.from_file) to catch type errors before loading in .NET.

Example fix

// before (tokenizer.json)
"pre_tokenizer": { "type": "Metaspace", "add_prefix_space": "true" }
// after
"pre_tokenizer": { "type": "Metaspace", "add_prefix_space": true }

// C# — before CreateFromTokenizerJson
using var doc = JsonDocument.Parse(tokenizerJsonText);
var root = doc.RootElement;
if (root.TryGetProperty("pre_tokenizer", out var pt) &&
    pt.TryGetProperty("type", out var t) && t.GetString() == "Metaspace" &&
    pt.TryGetProperty("add_prefix_space", out var aps) &&
    aps.ValueKind is not (JsonValueKind.True or JsonValueKind.False))
{
    throw new InvalidDataException("pre_tokenizer.add_prefix_space must be a JSON boolean.");
}

static bool IsValidAddPrefixSpace(JsonElement e) =>
    e.ValueKind == JsonValueKind.True || e.ValueKind == JsonValueKind.False;

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) { /* log ex.Message; fix tokenizer.json pre_tokenizer types */ }

When it happens

Trigger: Calling SentencePieceTokenizer.CreateFromTokenizerJson (or the code path that parses a tokenizer.json pre_tokenizer) where pre_tokenizer.type is 'Metaspace' and pre_tokenizer.add_prefix_space is present but not a JSON boolean — e.g. "add_prefix_space": "true" (string) or 1 (number).

Common situations: Hand-edited or model-converted tokenizer.json files; exporting tokenizers from a Python pipeline where add_prefix_space was serialized as a string; older Hugging Face tokenizers versions or third-party conversion tools that emitted a differently-typed add_prefix_space.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• The pre_tokenizer 'replacement' must be a string.
• The Metaspace 'replacement' '{replacement}' is not supported
• The model type is not Bpe.
• Normalization '{modelProto.NormalizerSpec.Name}' is not supp
• ArgumentNullException

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/cdd3cc71bb3ab7f3. Report an issue: GitHub.