{"record":{"id":"31ec03838eccaca0","repo":"dotnet/machinelearning","slug":"the-pre-tokenizer-replacement-must-be-a-string","errorCode":null,"errorMessage":"The pre_tokenizer 'replacement' must be a string.","messagePattern":"The pre_tokenizer 'replacement' must be a string\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":1237,"sourceCode":"\n            string? type = GetStringOrNull(preTokenizer, \"type\");\n            if (string.Equals(type, \"Metaspace\", StringComparison.OrdinalIgnoreCase))\n            {\n                if (preTokenizer.TryGetProperty(\"add_prefix_space\", out JsonElement addPrefixElement))\n                {\n                    if (addPrefixElement.ValueKind != JsonValueKind.True && addPrefixElement.ValueKind != JsonValueKind.False)\n                    {\n                        throw new InvalidDataException(\"The pre_tokenizer 'add_prefix_space' must be a boolean.\");\n                    }\n\n                    addDummyPrefix = addPrefixElement.GetBoolean();\n                }\n\n                if (preTokenizer.TryGetProperty(\"replacement\", out JsonElement replacementElement))\n                {\n                    if (replacementElement.ValueKind != JsonValueKind.String && replacementElement.ValueKind != JsonValueKind.Null)\n                    {\n                        throw new InvalidDataException(\"The pre_tokenizer 'replacement' must be a string.\");\n                    }\n\n                    // HF Metaspace's 'replacement' is the actual whitespace marker character. The SentencePiece model\n                    // only supports U+2581 ('▁'); reject any other marker rather than silently not escaping spaces.\n                    string? replacement = replacementElement.GetString();\n                    if (replacement is not null && replacement != \"\\u2581\") // U+2581 LOWER ONE EIGHTH BLOCK (▁)\n                    {\n                        throw new NotSupportedException(\n                            $\"The Metaspace 'replacement' '{replacement}' is not supported; only U+2581 ('\\u2581') is supported.\");\n                    }\n\n                    escapeWhiteSpaces = true;\n                }\n\n                if (preTokenizer.TryGetProperty(\"prepend_scheme\", out JsonElement prependSchemeElement))\n                {\n                    string? scheme = prependSchemeElement.ValueKind == JsonValueKind.String ? prependSchemeElement.GetString() : null;\n                    // \"never\" suppresses the dummy prefix; \"always\"/\"first\" keep the default (true)","sourceCodeStart":1219,"sourceCodeEnd":1255,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L1219-L1255","documentation":"The Metaspace pre-tokenizer's 'replacement' property must be either a JSON string or null. It specifies the whitespace marker character used when replacing spaces; a non-string, non-null JSON value (number, boolean, object, array) cannot be read as a character, so SentencePieceTokenizer rejects the tokenizer.json with InvalidDataException.","triggerScenarios":"Loading a tokenizer.json where pre_tokenizer is Metaspace and pre_tokenizer.replacement is present with a JSON kind other than String or Null — e.g. \"replacement\": 9601 or true.","commonSituations":"Corrupted or hand-edited tokenizer.json files; conversion tools that wrote the replacement marker's codepoint as a number; copy/paste edits that dropped the quotes around \"▁\".","solutions":["Edit tokenizer.json so pre_tokenizer.replacement is a JSON string, e.g. \"replacement\": \"▁\" (U+2581).","Use null (or omit the property) if no replacement marker is needed.","Regenerate tokenizer.json with the Hugging Face tokenizers save_pretrained flow so the Metaspace options keep correct types.","Lint/parse the file with Python tokenizers (Tokenizers.from_file) before loading in .NET to locate malformed fields."],"exampleFix":"// before (tokenizer.json)\n\"pre_tokenizer\": { \"type\": \"Metaspace\", \"replacement\": 9601 }\n// after\n\"pre_tokenizer\": { \"type\": \"Metaspace\", \"replacement\": \"▁\" }","handlingStrategy":"validation","validationCode":"// C# — check replacement type before loading\nif (preTokenizer.TryGetProperty(\"replacement\", out var r) &&\n    r.ValueKind is not (JsonValueKind.String or JsonValueKind.Null))\n{\n    throw new InvalidDataException(\"pre_tokenizer.replacement must be a JSON string or null.\");\n}","typeGuard":"static bool IsValidReplacement(JsonElement e) =>\n    e.ValueKind == JsonValueKind.String || e.ValueKind == JsonValueKind.Null;","tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) { /* inspect pre_tokenizer.replacement in tokenizer.json */ }","preventionTips":["Quote the replacement marker in tokenizer.json: \"replacement\": \"▁\".","Regenerate tokenizer artifacts from source models rather than editing by hand.","Diff converted tokenizer.json against the original export to spot type drift.","Run Tokenizers.from_file in Python as a pre-flight validation."],"tags":["tokenizers","sentencepiece","json-validation","configuration"],"backgroundTag":"invalid-config-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}