dotnet/machinelearning · error · NotSupportedException

The tokenizer.json pre_tokenizer type '{type ?? "<missing>"}

Error message

The tokenizer.json pre_tokenizer type '{type ?? "<missing>"}' is not supported; only Metaspace, WhitespaceSplit, Whitespace, and Sequence are handled.

What it means

SentencePieceTokenizer's constructor validates the tokenizer.json pre_tokenizer via ValidatePreTokenizer and only supports Metaspace, WhitespaceSplit, Whitespace, and Sequence (of those). Any other type — or a missing 'type' field — means the library cannot reproduce the pre-tokenization behavior, so NotSupportedException is thrown at load time.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:1209

                string.Equals(type, "Whitespace", StringComparison.OrdinalIgnoreCase))
            {
                return;
            }

            if (string.Equals(type, "Sequence", StringComparison.OrdinalIgnoreCase))
            {
                if (preTokenizer.TryGetProperty("pretokenizers", out JsonElement pretokenizers) &&
                    pretokenizers.ValueKind == JsonValueKind.Array)
                {
                    foreach (JsonElement inner in pretokenizers.EnumerateArray())
                    {
                        ValidatePreTokenizer(inner);
                    }
                }
                return;
            }

            throw new NotSupportedException(
                $"The tokenizer.json pre_tokenizer type '{type ?? "<missing>"}' is not supported; only Metaspace, WhitespaceSplit, Whitespace, and Sequence are handled.");
        }

        private static void ExtractMetaspaceSettings(JsonElement preTokenizer, ref bool addDummyPrefix, ref bool escapeWhiteSpaces)
        {
            if (preTokenizer.ValueKind != JsonValueKind.Object)
            {
                return;
            }

            string? type = GetStringOrNull(preTokenizer, "type");
            if (string.Equals(type, "Metaspace", StringComparison.OrdinalIgnoreCase))
            {
                if (preTokenizer.TryGetProperty("add_prefix_space", out JsonElement addPrefixElement))
                {
                    if (addPrefixElement.ValueKind != JsonValueKind.True && addPrefixElement.ValueKind != JsonValueKind.False)
                    {
                        throw new InvalidDataException("The pre_tokenizer 'add_prefix_space' must be a boolean.");

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use a tokenizer.json whose pre_tokenizer is Metaspace, WhitespaceSplit, Whitespace, or a Sequence of only those (SentencePiece tokenizers normally use Metaspace)
2. Convert the model/tokenizer to a SentencePiece-compatible one, or use a different Microsoft.ML.Tokenizers model class that supports the type
3. Ensure pre_tokenizer object includes a string 'type' field; add "type": "Metaspace" if it was lost in editing
4. Upgrade Microsoft.ML.Tokenizers to the latest version in case support for the type was added, otherwise catch NotSupportedException and pre-split text yourself

Example fix

// before (tokenizer.json)
"pre_tokenizer": {"type": "ByteLevel"}
// after
"pre_tokenizer": {"type": "Metaspace", "add_prefix_space": true, "replacement": "▁"}

var type = doc.RootElement.TryGetProperty("pre_tokenizer", out var pt) && pt.TryGetProperty("type", out var t)
    ? t.GetString() : null;
string[] supported = { "Metaspace", "WhitespaceSplit", "Whitespace", "Sequence" };
if (type is null || !supported.Contains(type)) throw new NotSupportedException(type);

static bool IsSupportedPreTokenizer(JsonElement root) =>
    root.TryGetProperty("pre_tokenizer", out var pt) &&
    pt.ValueKind == JsonValueKind.Object &&
    pt.TryGetProperty("type", out var t) && t.ValueKind == JsonValueKind.String &&
    t.GetString() is "Metaspace" or "WhitespaceSplit" or "Whitespace" or "Sequence";

try { tok = SentencePieceTokenizer.Create(modelStream, vocabStream); }
catch (NotSupportedException ex) when (ex.Message.Contains("pre_tokenizer"))
{ /* use a SentencePiece tokenizer.json or pre-split text manually */ }

When it happens

Trigger: Loading a tokenizer.json whose pre_tokenizer.type is ByteLevel, Punctuation, Digits, Split, etc.; or a pre_tokenizer object with no 'type' property; or a Sequence wrapping an unsupported inner type (ValidatePreTokenizer recurses into each element).

Common situations: Pointing SentencePieceTokenizer at a GPT-2/ByteLevel tokenizer.json instead of a SentencePiece one; tokenizer.json saved with new pre-tokenizer types added in newer HuggingFace tokenizers releases; hand-built configs missing the 'type' key.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• tokenizer.json post_processor templates with more than one s
• tokenizer.json post_processor template does not contain a se
• Too many pre-tokenizers provided. Maximum is {MaxPreTokenize
• Pre-tokenizer cannot be null.
• Failed to decode token ids

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e7d20d9c3c9943ba. Report an issue: GitHub.