{"record":{"id":"670955f46845c648","repo":"dotnet/machinelearning","slug":"trie-data-size-exceeds-the-input-blob-size","errorCode":null,"errorMessage":"Trie data size exceeds the input blob size.","messagePattern":"Trie data size exceeds the input blob size\\.","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizer.cs","lineNumber":588,"sourceCode":"\n            if (blob.Length <= sizeof(uint))\n            {\n                throw new ArgumentException(\"Blob for normalization rule is broken.\");\n            }\n\n            fixed (byte* pBlob = blob)\n            {\n                trieBlobSize = *(uint*)pBlob;\n            }\n\n            if (!BitConverter.IsLittleEndian)\n            {\n                trieBlobSize = Helpers.Swap32(trieBlobSize);\n            }\n\n            if (trieBlobSize >= blob.Length)\n            {\n                throw new ArgumentException(\"Trie data size exceeds the input blob size.\");\n            }\n\n            blob = blob.Slice(sizeof(uint));\n\n            if (!BitConverter.IsLittleEndian)\n            {\n                fixed (byte* pBlob = blob)\n                {\n                    uint* data = (uint*)pBlob;\n\n                    // Perform necessary operations for Big Endian\n                    for (int i = 0; i < trieBlobSize / 4; ++i)\n                    {\n                        data[i] = Helpers.Swap32(data[i]);\n                    }\n                }\n            }\n","sourceCodeStart":570,"sourceCodeEnd":606,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizer.cs#L570-L606","documentation":"After reading the 4-byte trie size header (with endianness swap if needed), the blob is sliced into a trie section and a normalized-bytes section. If the declared trieBlobSize is not strictly smaller than the blob length, there is no room left for the normalization data, indicating a corrupt or mismatched blob, so this ArgumentException is thrown.","triggerScenarios":"Passing a precompiled_charsmap blob where the header-declared trie size covers the whole blob (or exceeds it) — e.g. bytes from a different proto field, concatenated incorrectly, or built with wrong endianness on serialization.","commonSituations":"Reimplementing the SentencePiece charsmap decoder with the wrong byte layout; splicing trie bytes without the 4-byte length prefix; mixing little/big-endian exports.","solutions":["Verify the blob layout: [4-byte trie size][trie DoubleArray units][normalized bytes] and that trie size < total length.","Re-extract the blob directly from normalizer_spec.precompiled_charsmap of the .model proto.","Confirm consistent endianness when the blob was produced by custom code (helpers apply Swap32 when needed)."],"exampleFix":"// before: trie bytes written without reserving the size header\nvar blob = trieBytes.Concat(normalizedBytes).ToArray();\n// after\nvar blob = BitConverter.GetBytes(trieBytes.Length).Concat(trieBytes).Concat(normalizedBytes).ToArray();","handlingStrategy":"validation","validationCode":"bool BlobLayoutOk(byte[] blob) {\n    if (blob.Length <= 4) return false;\n    uint trieSize = BitConverter.ToUInt32(blob, 0);\n    return trieSize > 0 && trieSize < blob.Length;\n}","typeGuard":"bool HasConsistentTrieSize(ReadOnlySpan<byte> blob) => blob.Length > sizeof(uint) && BitConverter.ToUInt32(blob) < (uint)blob.Length;","tryCatchPattern":"try { Decode(blob); }\ncatch (ArgumentException ex) when (ex.Message == \"Trie data size exceeds the input blob size.\") {\n    // rebuild the blob with the correct [size][trie][normalized] layout\n}","preventionTips":["Follow the exact layout: 4-byte trie size, then trie units, then normalized bytes","Keep endianness consistent between writer and reader","Unit-test custom charsmap serialization against a known-good SentencePiece model"],"tags":["sentencepiece","binary-data","size-limit","tokenizer"],"backgroundTag":"value-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}