dotnet/machinelearning · error · ArgumentException

Trie data size exceeds the input blob size.

Error message

Trie data size exceeds the input blob size.

What it means

After reading the 4-byte trie size header (with endianness swap if needed), the blob is sliced into a trie section and a normalized-bytes section. If the declared trieBlobSize is not strictly smaller than the blob length, there is no room left for the normalization data, indicating a corrupt or mismatched blob, so this ArgumentException is thrown.

Source

Thrown at src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizer.cs:588

            if (blob.Length <= sizeof(uint))
            {
                throw new ArgumentException("Blob for normalization rule is broken.");
            }

            fixed (byte* pBlob = blob)
            {
                trieBlobSize = *(uint*)pBlob;
            }

            if (!BitConverter.IsLittleEndian)
            {
                trieBlobSize = Helpers.Swap32(trieBlobSize);
            }

            if (trieBlobSize >= blob.Length)
            {
                throw new ArgumentException("Trie data size exceeds the input blob size.");
            }

            blob = blob.Slice(sizeof(uint));

            if (!BitConverter.IsLittleEndian)
            {
                fixed (byte* pBlob = blob)
                {
                    uint* data = (uint*)pBlob;

                    // Perform necessary operations for Big Endian
                    for (int i = 0; i < trieBlobSize / 4; ++i)
                    {
                        data[i] = Helpers.Swap32(data[i]);
                    }
                }
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Verify the blob layout: [4-byte trie size][trie DoubleArray units][normalized bytes] and that trie size < total length.
2. Re-extract the blob directly from normalizer_spec.precompiled_charsmap of the .model proto.
3. Confirm consistent endianness when the blob was produced by custom code (helpers apply Swap32 when needed).

Example fix

// before: trie bytes written without reserving the size header
var blob = trieBytes.Concat(normalizedBytes).ToArray();
// after
var blob = BitConverter.GetBytes(trieBytes.Length).Concat(trieBytes).Concat(normalizedBytes).ToArray();

bool BlobLayoutOk(byte[] blob) {
    if (blob.Length <= 4) return false;
    uint trieSize = BitConverter.ToUInt32(blob, 0);
    return trieSize > 0 && trieSize < blob.Length;
}

bool HasConsistentTrieSize(ReadOnlySpan<byte> blob) => blob.Length > sizeof(uint) && BitConverter.ToUInt32(blob) < (uint)blob.Length;

try { Decode(blob); }
catch (ArgumentException ex) when (ex.Message == "Trie data size exceeds the input blob size.") {
    // rebuild the blob with the correct [size][trie][normalized] layout
}

When it happens

Trigger: Passing a precompiled_charsmap blob where the header-declared trie size covers the whole blob (or exceeds it) — e.g. bytes from a different proto field, concatenated incorrectly, or built with wrong endianness on serialization.

Common situations: Reimplementing the SentencePiece charsmap decoder with the wrong byte layout; splicing trie bytes without the 4-byte length prefix; mixing little/big-endian exports.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Blob for normalization rule is broken.
• The tokenizer.json normalizer 'precompiled_charsmap' is not
• Failed to decode token ids
• throw new ArgumentNullException(nameof(vocabStream));
• The beginning of sentence token '{beginningOfSentenceToken}'

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/670955f46845c648. Report an issue: GitHub.