{"record":{"id":"cdd3cc71bb3ab7f3","repo":"dotnet/machinelearning","slug":"the-pre-tokenizer-add-prefix-space-must-be-a-boo","errorCode":null,"errorMessage":"The pre_tokenizer 'add_prefix_space' must be a boolean.","messagePattern":"The pre_tokenizer 'add_prefix_space' must be a boolean\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":1227,"sourceCode":"            throw new NotSupportedException(\n                $\"The tokenizer.json pre_tokenizer type '{type ?? \"<missing>\"}' is not supported; only Metaspace, WhitespaceSplit, Whitespace, and Sequence are handled.\");\n        }\n\n        private static void ExtractMetaspaceSettings(JsonElement preTokenizer, ref bool addDummyPrefix, ref bool escapeWhiteSpaces)\n        {\n            if (preTokenizer.ValueKind != JsonValueKind.Object)\n            {\n                return;\n            }\n\n            string? type = GetStringOrNull(preTokenizer, \"type\");\n            if (string.Equals(type, \"Metaspace\", StringComparison.OrdinalIgnoreCase))\n            {\n                if (preTokenizer.TryGetProperty(\"add_prefix_space\", out JsonElement addPrefixElement))\n                {\n                    if (addPrefixElement.ValueKind != JsonValueKind.True && addPrefixElement.ValueKind != JsonValueKind.False)\n                    {\n                        throw new InvalidDataException(\"The pre_tokenizer 'add_prefix_space' must be a boolean.\");\n                    }\n\n                    addDummyPrefix = addPrefixElement.GetBoolean();\n                }\n\n                if (preTokenizer.TryGetProperty(\"replacement\", out JsonElement replacementElement))\n                {\n                    if (replacementElement.ValueKind != JsonValueKind.String && replacementElement.ValueKind != JsonValueKind.Null)\n                    {\n                        throw new InvalidDataException(\"The pre_tokenizer 'replacement' must be a string.\");\n                    }\n\n                    // HF Metaspace's 'replacement' is the actual whitespace marker character. The SentencePiece model\n                    // only supports U+2581 ('▁'); reject any other marker rather than silently not escaping spaces.\n                    string? replacement = replacementElement.GetString();\n                    if (replacement is not null && replacement != \"\\u2581\") // U+2581 LOWER ONE EIGHTH BLOCK (▁)\n                    {\n                        throw new NotSupportedException(","sourceCodeStart":1209,"sourceCodeEnd":1245,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L1209-L1245","documentation":"When building a SentencePieceTokenizer from a tokenizer.json, the Metaspace pre-tokenizer's 'add_prefix_space' option must be JSON true or false. The library reads this property to decide whether to prepend the ▁ dummy prefix to input text; any other JSON kind (string, number, null, object) cannot be interpreted as a boolean, so loading fails with InvalidDataException rather than guessing a default.","triggerScenarios":"Calling SentencePieceTokenizer.CreateFromTokenizerJson (or the code path that parses a tokenizer.json pre_tokenizer) where pre_tokenizer.type is 'Metaspace' and pre_tokenizer.add_prefix_space is present but not a JSON boolean — e.g. \"add_prefix_space\": \"true\" (string) or 1 (number).","commonSituations":"Hand-edited or model-converted tokenizer.json files; exporting tokenizers from a Python pipeline where add_prefix_space was serialized as a string; older Hugging Face tokenizers versions or third-party conversion tools that emitted a differently-typed add_prefix_space.","solutions":["Open tokenizer.json and change pre_tokenizer.add_prefix_space to a literal JSON boolean: true or false.","Remove the add_prefix_space property entirely if you want the library default behavior (it is only validated when present).","Regenerate tokenizer.json with the Hugging Face tokenizers library (save_pretrained) instead of hand-editing, ensuring the Metaspace pre-tokenizer options keep their native types.","Validate the tokenizer.json with the HF tokenizers Python library (Tokenizers.from_file) to catch type errors before loading in .NET."],"exampleFix":"// before (tokenizer.json)\n\"pre_tokenizer\": { \"type\": \"Metaspace\", \"add_prefix_space\": \"true\" }\n// after\n\"pre_tokenizer\": { \"type\": \"Metaspace\", \"add_prefix_space\": true }","handlingStrategy":"validation","validationCode":"// C# — before CreateFromTokenizerJson\nusing var doc = JsonDocument.Parse(tokenizerJsonText);\nvar root = doc.RootElement;\nif (root.TryGetProperty(\"pre_tokenizer\", out var pt) &&\n    pt.TryGetProperty(\"type\", out var t) && t.GetString() == \"Metaspace\" &&\n    pt.TryGetProperty(\"add_prefix_space\", out var aps) &&\n    aps.ValueKind is not (JsonValueKind.True or JsonValueKind.False))\n{\n    throw new InvalidDataException(\"pre_tokenizer.add_prefix_space must be a JSON boolean.\");\n}","typeGuard":"static bool IsValidAddPrefixSpace(JsonElement e) =>\n    e.ValueKind == JsonValueKind.True || e.ValueKind == JsonValueKind.False;","tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) { /* log ex.Message; fix tokenizer.json pre_tokenizer types */ }","preventionTips":["Never hand-edit tokenizer.json; regenerate via HF tokenizers save_pretrained.","Treat pre_tokenizer options as typed JSON, not strings ('true' != true).","Validate tokenizer.json with the Python tokenizers library before deployment.","Add a config lint step in CI that checks Metaspace option types."],"tags":["tokenizers","sentencepiece","json-validation","configuration"],"backgroundTag":"invalid-config-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}