dotnet/machinelearning · error · ArgumentException

Blob for normalization rule is broken.

Error message

Blob for normalization rule is broken.

What it means

DecodePrecompiledCharsMap unpacks the SentencePiece precompiled charsmap blob: it expects at least a 4-byte trie size header plus trie and normalized-data sections. If the blob is shorter than sizeof(uint)+1 bytes there is no usable header, so it throws this ArgumentException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizer.cs:573

                else
                {
                    if (normalized.Length <= normalizedIndex + 1)
                    {
                        Helpers.ArrayPoolGrow(ref normalized, ref poolArray, (normalizedIndex + 1) << 1);
                    }
                    normalized[normalizedIndex] = (byte)' ';
                    normalizedIndex++;
                }
            }
        }

        private unsafe void DecodePrecompiledCharsMap(ReadOnlySpan<byte> blob, out DoubleArrayUnit[]? trieBlob, out byte[]? normalized)
        {
            uint trieBlobSize = 0;

            if (blob.Length <= sizeof(uint))
            {
                throw new ArgumentException("Blob for normalization rule is broken.");
            }

            fixed (byte* pBlob = blob)
            {
                trieBlobSize = *(uint*)pBlob;
            }

            if (!BitConverter.IsLittleEndian)
            {
                trieBlobSize = Helpers.Swap32(trieBlobSize);
            }

            if (trieBlobSize >= blob.Length)
            {
                throw new ArgumentException("Trie data size exceeds the input blob size.");
            }

            blob = blob.Slice(sizeof(uint));

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use the full precompiled_charsmap from the original SentencePiece model proto (normalizer_spec.precompiled_charsmap).
2. Re-export the charsmap with sentencepiece_model_pb2 rather than hand-assembling bytes.
3. Check the base64 string length: a valid charsmap blob is typically tens of KB.

Example fix

// before: stub blob
var normalizer = new SentencePieceNormalizer(precompiledCharsMap: new byte[] { 1, 2, 3 }, ...);
// after: real blob from model proto
var normalizer = new SentencePieceNormalizer(precompiledCharsMap: model.NormalizerSpec.PrecompiledCharsmap.ToByteArray(), ...);

bool CharsmapBlobPlausible(byte[] blob) => blob is { Length: > 1024 }; // real charsmaps are large; header alone needs >4 bytes

bool HasValidCharsmapHeader(ReadOnlySpan<byte> blob) => blob.Length > sizeof(uint) && BitConverter.ToUInt32(blob) > 0;

try { var norm = new SentencePieceNormalizer(blob, ...); }
catch (ArgumentException ex) when (ex.Message == "Blob for normalization rule is broken.") {
    // re-extract the blob from the SentencePiece model proto
}

When it happens

Trigger: Loading a SentencePiece normalizer whose decoded precompiled_charsmap blob is empty or only 1-4 bytes — e.g. base64 decoded to a stub, corrupted file, or wrongly extracted blob.

Common situations: Charsmap blob built manually from spm_normalize export with wrong byte order/truncation; test fixtures with dummy values; partial file downloads.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Trie data size exceeds the input blob size.
• The tokenizer.json normalizer 'precompiled_charsmap' is not
• Too many pre-tokenizers provided. Maximum is {MaxPreTokenize
• failed to insert key: negative value
• Exception of type 'System.ArgumentException' was thrown.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/48ed12bffa6b5930. Report an issue: GitHub.