{"record":{"id":"7e9c9cae54b63f69","repo":"dotnet/machinelearning","slug":"the-precompiled-normalizer-precompiled-charsmap","errorCode":null,"errorMessage":"The Precompiled normalizer 'precompiled_charsmap' must be a string.","messagePattern":"The Precompiled normalizer 'precompiled_charsmap' must be a string\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs","lineNumber":156,"sourceCode":"                case \"Sequence\":\n                    var children = new List<SentencePieceNormalizationStep>();\n                    if (normalizer.TryGetProperty(\"normalizers\", out JsonElement steps) && steps.ValueKind == JsonValueKind.Array)\n                    {\n                        foreach (JsonElement step in steps.EnumerateArray())\n                        {\n                            children.Add(Build(step));\n                        }\n                    }\n                    return new SequenceStep(children);\n\n                case \"Precompiled\":\n                    {\n                        string? charsMap = null;\n                        if (normalizer.TryGetProperty(\"precompiled_charsmap\", out JsonElement mapElement))\n                        {\n                            if (mapElement.ValueKind != JsonValueKind.String && mapElement.ValueKind != JsonValueKind.Null)\n                            {\n                                throw new InvalidDataException(\"The Precompiled normalizer 'precompiled_charsmap' must be a string.\");\n                            }\n\n                            charsMap = mapElement.GetString();\n                        }\n\n                        return new PrecompiledStep(string.IsNullOrEmpty(charsMap) ? default : DecodePrecompiledCharsMap(charsMap!));\n                    }\n\n                case \"Replace\":\n                    return ReplaceStep.Create(normalizer);\n\n                case \"Strip\":\n                    return new StripStep(\n                        stripLeft: !normalizer.TryGetProperty(\"strip_left\", out JsonElement left) || left.ValueKind != JsonValueKind.False,\n                        stripRight: !normalizer.TryGetProperty(\"strip_right\", out JsonElement right) || right.ValueKind != JsonValueKind.False);\n\n                case \"Lowercase\":\n                    return LowercaseStep.Instance;","sourceCodeStart":138,"sourceCodeEnd":174,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs#L138-L174","documentation":"When the tokenizer.json normalizer type is 'Precompiled' (SentencePiece's precompiled charsmap), the 'precompiled_charsmap' property, if present, must be a JSON string (or null) containing base64 data. Any other JSON kind throws InvalidDataException.","triggerScenarios":"Loading a tokenizer.json whose normalizer is {\"type\": \"Precompiled\", \"precompiled_charsmap\": <non-string>} — e.g. the charsmap serialized as an array of bytes or a number instead of a base64 string.","commonSituations":"tokenizer.json produced by a tool that serializes charsmap differently; manual conversion of a sentencepiece model proto where the bytes were emitted as an array; corrupted file.","solutions":["Re-export the tokenizer.json with Hugging Face tokenizers so precompiled_charsmap is a base64 string","Convert the charsmap bytes to base64 in the JSON if hand-authoring","Remove the precompiled_charsmap property if no precompilation is needed (it is treated as default)"],"exampleFix":"// before\n\"precompiled_charsmap\": [210, 128, 1]\n// after\n\"precompiled_charsmap\": \"4oKwAQA=\"","handlingStrategy":"validation","validationCode":"if (normalizer.TryGetProperty(\"precompiled_charsmap\", out var m) && m.ValueKind != JsonValueKind.String && m.ValueKind != JsonValueKind.Null)\n    throw new InvalidDataException(\"precompiled_charsmap must be a base64 string\");","typeGuard":"bool IsValidCharsMap(JsonElement e) => !e.TryGetProperty(\"precompiled_charsmap\", out var m) || m.ValueKind == JsonValueKind.String || m.ValueKind == JsonValueKind.Null;","tryCatchPattern":"try { LoadTokenizer(tokenizerJson); } catch (InvalidDataException ex) when (ex.Message.Contains(\"precompiled_charsmap\")) { log.LogError(ex, \"Invalid Precompiled normalizer charsmap\"); }","preventionTips":["Always export precompiled_charsmap as a base64 string","Convert sentencepiece proto bytes to base64 when converting models manually","Diff generated tokenizer.json against a known-good export"],"tags":["tokenizers","json","sentencepiece"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}