{"record":{"id":"e7d20d9c3c9943ba","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-pre-tokenizer-type-type","errorCode":null,"errorMessage":"The tokenizer.json pre_tokenizer type '{type ?? \"<missing>\"}' is not supported; only Metaspace, WhitespaceSplit, Whitespace, and Sequence are handled.","messagePattern":"The tokenizer\\.json pre_tokenizer type '(.+?)' is not supported; only Metaspace, WhitespaceSplit, Whitespace, and Sequence are handled\\.","errorType":"validation","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":1209,"sourceCode":"                string.Equals(type, \"Whitespace\", StringComparison.OrdinalIgnoreCase))\n            {\n                return;\n            }\n\n            if (string.Equals(type, \"Sequence\", StringComparison.OrdinalIgnoreCase))\n            {\n                if (preTokenizer.TryGetProperty(\"pretokenizers\", out JsonElement pretokenizers) &&\n                    pretokenizers.ValueKind == JsonValueKind.Array)\n                {\n                    foreach (JsonElement inner in pretokenizers.EnumerateArray())\n                    {\n                        ValidatePreTokenizer(inner);\n                    }\n                }\n                return;\n            }\n\n            throw new NotSupportedException(\n                $\"The tokenizer.json pre_tokenizer type '{type ?? \"<missing>\"}' is not supported; only Metaspace, WhitespaceSplit, Whitespace, and Sequence are handled.\");\n        }\n\n        private static void ExtractMetaspaceSettings(JsonElement preTokenizer, ref bool addDummyPrefix, ref bool escapeWhiteSpaces)\n        {\n            if (preTokenizer.ValueKind != JsonValueKind.Object)\n            {\n                return;\n            }\n\n            string? type = GetStringOrNull(preTokenizer, \"type\");\n            if (string.Equals(type, \"Metaspace\", StringComparison.OrdinalIgnoreCase))\n            {\n                if (preTokenizer.TryGetProperty(\"add_prefix_space\", out JsonElement addPrefixElement))\n                {\n                    if (addPrefixElement.ValueKind != JsonValueKind.True && addPrefixElement.ValueKind != JsonValueKind.False)\n                    {\n                        throw new InvalidDataException(\"The pre_tokenizer 'add_prefix_space' must be a boolean.\");","sourceCodeStart":1191,"sourceCodeEnd":1227,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L1191-L1227","documentation":"SentencePieceTokenizer's constructor validates the tokenizer.json pre_tokenizer via ValidatePreTokenizer and only supports Metaspace, WhitespaceSplit, Whitespace, and Sequence (of those). Any other type — or a missing 'type' field — means the library cannot reproduce the pre-tokenization behavior, so NotSupportedException is thrown at load time.","triggerScenarios":"Loading a tokenizer.json whose pre_tokenizer.type is ByteLevel, Punctuation, Digits, Split, etc.; or a pre_tokenizer object with no 'type' property; or a Sequence wrapping an unsupported inner type (ValidatePreTokenizer recurses into each element).","commonSituations":"Pointing SentencePieceTokenizer at a GPT-2/ByteLevel tokenizer.json instead of a SentencePiece one; tokenizer.json saved with new pre-tokenizer types added in newer HuggingFace tokenizers releases; hand-built configs missing the 'type' key.","solutions":["Use a tokenizer.json whose pre_tokenizer is Metaspace, WhitespaceSplit, Whitespace, or a Sequence of only those (SentencePiece tokenizers normally use Metaspace)","Convert the model/tokenizer to a SentencePiece-compatible one, or use a different Microsoft.ML.Tokenizers model class that supports the type","Ensure pre_tokenizer object includes a string 'type' field; add \"type\": \"Metaspace\" if it was lost in editing","Upgrade Microsoft.ML.Tokenizers to the latest version in case support for the type was added, otherwise catch NotSupportedException and pre-split text yourself"],"exampleFix":"// before (tokenizer.json)\n\"pre_tokenizer\": {\"type\": \"ByteLevel\"}\n// after\n\"pre_tokenizer\": {\"type\": \"Metaspace\", \"add_prefix_space\": true, \"replacement\": \"▁\"}","handlingStrategy":"try-catch","validationCode":"var type = doc.RootElement.TryGetProperty(\"pre_tokenizer\", out var pt) && pt.TryGetProperty(\"type\", out var t)\n    ? t.GetString() : null;\nstring[] supported = { \"Metaspace\", \"WhitespaceSplit\", \"Whitespace\", \"Sequence\" };\nif (type is null || !supported.Contains(type)) throw new NotSupportedException(type);","typeGuard":"static bool IsSupportedPreTokenizer(JsonElement root) =>\n    root.TryGetProperty(\"pre_tokenizer\", out var pt) &&\n    pt.ValueKind == JsonValueKind.Object &&\n    pt.TryGetProperty(\"type\", out var t) && t.ValueKind == JsonValueKind.String &&\n    t.GetString() is \"Metaspace\" or \"WhitespaceSplit\" or \"Whitespace\" or \"Sequence\";","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(modelStream, vocabStream); }\ncatch (NotSupportedException ex) when (ex.Message.Contains(\"pre_tokenizer\"))\n{ /* use a SentencePiece tokenizer.json or pre-split text manually */ }","preventionTips":["Confirm the tokenizer.json is SentencePiece-based (Metaspace pre-tokenizer) before using SentencePieceTokenizer","Check the pre_tokenizer.type field when migrating tokenizers between libraries","Handle missing 'type' keys defensively in generated configs","Pin Microsoft.ML.Tokenizers versions and review release notes for newly supported pre-tokenizer types"],"tags":["tokenizer","pre-tokenizer","unsupported-feature","compatibility"],"backgroundTag":"unsupported-operation","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}