{"record":{"id":"fc2798cafac471b2","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-normalizer-precompiled-charsma","errorCode":null,"errorMessage":"The tokenizer.json normalizer 'precompiled_charsmap' is not valid base64.","messagePattern":"The tokenizer\\.json normalizer 'precompiled_charsmap' is not valid base64\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs","lineNumber":219,"sourceCode":"                    return NmtStep.Instance;\n\n                default:\n                    throw new NotSupportedException(\n                        $\"Unigram normalizer type '{type ?? \"<missing>\"}' is not supported when loading a tokenizer.json with content-modifying normalizer steps.\");\n            }\n        }\n\n        // Decodes a base64 'precompiled_charsmap' value, surfacing malformed input as InvalidDataException so callers\n        // get a consistent, diagnostic failure for bad tokenizer.json files instead of a raw FormatException.\n        internal static byte[] DecodePrecompiledCharsMap(string base64)\n        {\n            try\n            {\n                return Convert.FromBase64String(base64);\n            }\n            catch (FormatException ex)\n            {\n                throw new InvalidDataException(\"The tokenizer.json normalizer 'precompiled_charsmap' is not valid base64.\", ex);\n            }\n        }\n\n        // Mirrors SentencePieceTokenizer.ReplaceCollapsesSpaces: a Replace whose Regex matches runs of spaces.\n        private static bool ReplaceIsWhitespaceCollapse(JsonElement replace)\n        {\n            if (!replace.TryGetProperty(\"pattern\", out JsonElement patternElement) ||\n                patternElement.ValueKind != JsonValueKind.Object ||\n                !patternElement.TryGetProperty(\"Regex\", out JsonElement regexElement) ||\n                regexElement.ValueKind != JsonValueKind.String)\n            {\n                return false;\n            }\n\n            switch (regexElement.GetString())\n            {\n                case \" {2,}\":\n                case \" +\":","sourceCodeStart":201,"sourceCodeEnd":237,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs#L201-L237","documentation":"When loading a tokenizer.json that uses the 'precompiled_charsmap' normalizer (typical of SentencePiece models), the library decodes the charsmap blob from a base64 string via Convert.FromBase64String. If the string is not valid base64 (wrong characters, padding, or whitespace), a FormatException is caught and rethrown as this InvalidDataException. The precompiled charsmap is required for the normalizer to run, so loading cannot continue.","triggerScenarios":"Calling Tokenizer.Create / SentencePieceNormalizer with a tokenizer.json whose normalizer.precompiled_charsmap value is not valid base64 — e.g. hand-edited JSON, truncated file, or a charsmap exported from a non-HuggingFace tool.","commonSituations":"Manually editing or re-serializing tokenizer.json and corrupting the long base64 string; downloading/truncating the file; using a tokenizer.json generated by an older or nonstandard exporter; string escaping mangled the value.","solutions":["Re-download or regenerate tokenizer.json from the original model (e.g. transformers/AutoTokenizer.convert_slow_tokenizer) instead of hand-editing it.","Validate the string with Convert.FromBase64String (or Convert.TryFromBase64String) before loading to pinpoint corruption.","If only the normalizer is corrupted, replace the normalizer with a null/identity normalizer in tokenizer.json, accepting the normalization loss."],"exampleFix":"// before: hand-edited, corrupted charsmap\n\"precompiled_charsmap\": \"eJyrVspMUbIyszQ...\" (truncated)\n// after: fresh file from the model repo\n\"precompiled_charsmap\": \"AAEAAOAKEEVWRl=...\" (complete, valid base64)","handlingStrategy":"try-catch","validationCode":"bool IsValidBase64(string s) => !string.IsNullOrEmpty(s) && Convert.TryFromBase64String(s, new byte[s.Length * 3 / 4 + 3], out _);","typeGuard":"bool IsValidBase64(string? s) => s is not null && s.Length % 4 == 0 && Regex.IsMatch(s, \"^[A-Za-z0-9+/]*={0,2}$\");","tryCatchPattern":"try { var tokenizer = Tokenizer.Create(model); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"precompiled_charsmap\")) {\n    // reload a pristine tokenizer.json or fall back to a normalizer-less model\n}","preventionTips":["Never hand-edit the precompiled_charsmap base64 string","Verify file integrity (hash) after download","Load tokenizer.json with the official tokenizers library first as a smoke test"],"tags":["tokenizer","base64","sentencepiece","invalid-data"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}