{"record":{"id":"e9de060c133e04a2","repo":"dotnet/machinelearning","slug":"the-metaspace-replacement-replacement-is-not","errorCode":null,"errorMessage":"The Metaspace 'replacement' '{replacement}' is not supported; only U+2581 ('▁') is supported.","messagePattern":"The Metaspace 'replacement' '(.+?)' is not supported; only U\\+2581 \\('▁'\\) is supported\\.","errorType":"validation","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":1245,"sourceCode":"                        throw new InvalidDataException(\"The pre_tokenizer 'add_prefix_space' must be a boolean.\");\n                    }\n\n                    addDummyPrefix = addPrefixElement.GetBoolean();\n                }\n\n                if (preTokenizer.TryGetProperty(\"replacement\", out JsonElement replacementElement))\n                {\n                    if (replacementElement.ValueKind != JsonValueKind.String && replacementElement.ValueKind != JsonValueKind.Null)\n                    {\n                        throw new InvalidDataException(\"The pre_tokenizer 'replacement' must be a string.\");\n                    }\n\n                    // HF Metaspace's 'replacement' is the actual whitespace marker character. The SentencePiece model\n                    // only supports U+2581 ('▁'); reject any other marker rather than silently not escaping spaces.\n                    string? replacement = replacementElement.GetString();\n                    if (replacement is not null && replacement != \"\\u2581\") // U+2581 LOWER ONE EIGHTH BLOCK (▁)\n                    {\n                        throw new NotSupportedException(\n                            $\"The Metaspace 'replacement' '{replacement}' is not supported; only U+2581 ('\\u2581') is supported.\");\n                    }\n\n                    escapeWhiteSpaces = true;\n                }\n\n                if (preTokenizer.TryGetProperty(\"prepend_scheme\", out JsonElement prependSchemeElement))\n                {\n                    string? scheme = prependSchemeElement.ValueKind == JsonValueKind.String ? prependSchemeElement.GetString() : null;\n                    // \"never\" suppresses the dummy prefix; \"always\"/\"first\" keep the default (true)\n                    if (string.Equals(scheme, \"never\", StringComparison.OrdinalIgnoreCase))\n                    {\n                        addDummyPrefix = false;\n                    }\n                }\n            }\n            else if (string.Equals(type, \"Sequence\", StringComparison.OrdinalIgnoreCase) &&\n                     preTokenizer.TryGetProperty(\"pretokenizers\", out JsonElement preTokenizersElement) &&","sourceCodeStart":1227,"sourceCodeEnd":1263,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L1227-L1263","documentation":"The SentencePiece model format only supports U+2581 ('▁', lower one eighth block) as the Metaspace whitespace marker. If tokenizer.json's Metaspace pre_tokenizer specifies a different 'replacement' character, Microsoft.ML.Tokenizers throws NotSupportedException instead of silently failing to escape spaces during encoding.","triggerScenarios":"Loading a tokenizer.json whose Metaspace pre_tokenizer.replacement is a non-null string other than \"▁\" — e.g. \"replacement\": \"Ġ\" (as in GPT-style byte-level BPE Metaspace variants) or \"_\".","commonSituations":"Tokenizers exported from byte-level pipelines that use different space markers; mixing pre_tokenizer configs across model families; hand-porting a BPE tokenizer.json to be loaded as SentencePiece.","solutions":["Set pre_tokenizer.replacement to \"▁\" (U+2581) in tokenizer.json.","Set replacement to null or remove it to fall back to the model's default behavior.","Verify the tokenizer actually belongs to a SentencePiece model; if it uses a different space marker, load it with the appropriate tokenizer type (e.g. BPE) rather than SentencePieceTokenizer.","Regenerate the tokenizer artifact from the original SentencePiece training run so the standard ▁ marker is produced."],"exampleFix":"// before (tokenizer.json)\n\"pre_tokenizer\": { \"type\": \"Metaspace\", \"replacement\": \"_\" }\n// after\n\"pre_tokenizer\": { \"type\": \"Metaspace\", \"replacement\": \"▁\" }","handlingStrategy":"validation","validationCode":"// C# — verify replacement marker before loading\nif (preTokenizer.TryGetProperty(\"replacement\", out var r) &&\n    r.ValueKind == JsonValueKind.String &&\n    r.GetString() is string s && s != \"\\u2581\")\n{\n    throw new NotSupportedException($\"Metaspace replacement '{s}' unsupported; use U+2581.\");\n}","typeGuard":"static bool IsSupportedReplacement(string? r) => r is null || r == \"\\u2581\";","tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (NotSupportedException ex) { /* the model is not SentencePiece-compatible; use correct tokenizer type */ }","preventionTips":["Only load SentencePiece-trained tokenizers with SentencePieceTokenizer.","Check the space marker family: Ġ-style markers indicate byte-level BPE, not SentencePiece.","Keep replacement at U+2581 for all Metaspace pre-tokenizers used with this library.","Document the source model type alongside each tokenizer artifact."],"tags":["tokenizers","sentencepiece","unsupported-value","configuration"],"backgroundTag":"unsupported-config-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}