dotnet/machinelearning · error · InvalidDataException

The tokenizer.json normalizer 'precompiled_charsmap' is not

Error message

The tokenizer.json normalizer 'precompiled_charsmap' is not valid base64.

What it means

When loading a tokenizer.json that uses the 'precompiled_charsmap' normalizer (typical of SentencePiece models), the library decodes the charsmap blob from a base64 string via Convert.FromBase64String. If the string is not valid base64 (wrong characters, padding, or whitespace), a FormatException is caught and rethrown as this InvalidDataException. The precompiled charsmap is required for the normalizer to run, so loading cannot continue.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizationStep.cs:219

                    return NmtStep.Instance;

                default:
                    throw new NotSupportedException(
                        $"Unigram normalizer type '{type ?? "<missing>"}' is not supported when loading a tokenizer.json with content-modifying normalizer steps.");
            }
        }

        // Decodes a base64 'precompiled_charsmap' value, surfacing malformed input as InvalidDataException so callers
        // get a consistent, diagnostic failure for bad tokenizer.json files instead of a raw FormatException.
        internal static byte[] DecodePrecompiledCharsMap(string base64)
        {
            try
            {
                return Convert.FromBase64String(base64);
            }
            catch (FormatException ex)
            {
                throw new InvalidDataException("The tokenizer.json normalizer 'precompiled_charsmap' is not valid base64.", ex);
            }
        }

        // Mirrors SentencePieceTokenizer.ReplaceCollapsesSpaces: a Replace whose Regex matches runs of spaces.
        private static bool ReplaceIsWhitespaceCollapse(JsonElement replace)
        {
            if (!replace.TryGetProperty("pattern", out JsonElement patternElement) ||
                patternElement.ValueKind != JsonValueKind.Object ||
                !patternElement.TryGetProperty("Regex", out JsonElement regexElement) ||
                regexElement.ValueKind != JsonValueKind.String)
            {
                return false;
            }

            switch (regexElement.GetString())
            {
                case " {2,}":
                case " +":

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Re-download or regenerate tokenizer.json from the original model (e.g. transformers/AutoTokenizer.convert_slow_tokenizer) instead of hand-editing it.
2. Validate the string with Convert.FromBase64String (or Convert.TryFromBase64String) before loading to pinpoint corruption.
3. If only the normalizer is corrupted, replace the normalizer with a null/identity normalizer in tokenizer.json, accepting the normalization loss.

Example fix

// before: hand-edited, corrupted charsmap
"precompiled_charsmap": "eJyrVspMUbIyszQ..." (truncated)
// after: fresh file from the model repo
"precompiled_charsmap": "AAEAAOAKEEVWRl=..." (complete, valid base64)

bool IsValidBase64(string s) => !string.IsNullOrEmpty(s) && Convert.TryFromBase64String(s, new byte[s.Length * 3 / 4 + 3], out _);

bool IsValidBase64(string? s) => s is not null && s.Length % 4 == 0 && Regex.IsMatch(s, "^[A-Za-z0-9+/]*={0,2}$");

try { var tokenizer = Tokenizer.Create(model); }
catch (InvalidDataException ex) when (ex.Message.Contains("precompiled_charsmap")) {
    // reload a pristine tokenizer.json or fall back to a normalizer-less model
}

When it happens

Trigger: Calling Tokenizer.Create / SentencePieceNormalizer with a tokenizer.json whose normalizer.precompiled_charsmap value is not valid base64 — e.g. hand-edited JSON, truncated file, or a charsmap exported from a non-HuggingFace tool.

Common situations: Manually editing or re-serializing tokenizer.json and corrupting the long base64 string; downloading/truncating the file; using a tokenizer.json generated by an older or nonstandard exporter; string escaping mangled the value.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• An 'added_tokens' entry must have a string 'content' and a n
• The tokenizer.json post_processor special token '{tokenName}
• The post-processor '{property}' token '{token}' with id {id}
• Replace normalizer has an invalid Regex pattern '{regexPatte
• Blob for normalization rule is broken.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/fc2798cafac471b2. Report an issue: GitHub.