{"record":{"id":"48ed12bffa6b5930","repo":"dotnet/machinelearning","slug":"blob-for-normalization-rule-is-broken","errorCode":null,"errorMessage":"Blob for normalization rule is broken.","messagePattern":"Blob for normalization rule is broken\\.","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizer.cs","lineNumber":573,"sourceCode":"                else\n                {\n                    if (normalized.Length <= normalizedIndex + 1)\n                    {\n                        Helpers.ArrayPoolGrow(ref normalized, ref poolArray, (normalizedIndex + 1) << 1);\n                    }\n                    normalized[normalizedIndex] = (byte)' ';\n                    normalizedIndex++;\n                }\n            }\n        }\n\n        private unsafe void DecodePrecompiledCharsMap(ReadOnlySpan<byte> blob, out DoubleArrayUnit[]? trieBlob, out byte[]? normalized)\n        {\n            uint trieBlobSize = 0;\n\n            if (blob.Length <= sizeof(uint))\n            {\n                throw new ArgumentException(\"Blob for normalization rule is broken.\");\n            }\n\n            fixed (byte* pBlob = blob)\n            {\n                trieBlobSize = *(uint*)pBlob;\n            }\n\n            if (!BitConverter.IsLittleEndian)\n            {\n                trieBlobSize = Helpers.Swap32(trieBlobSize);\n            }\n\n            if (trieBlobSize >= blob.Length)\n            {\n                throw new ArgumentException(\"Trie data size exceeds the input blob size.\");\n            }\n\n            blob = blob.Slice(sizeof(uint));","sourceCodeStart":555,"sourceCodeEnd":591,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Normalizer/SentencePieceNormalizer.cs#L555-L591","documentation":"DecodePrecompiledCharsMap unpacks the SentencePiece precompiled charsmap blob: it expects at least a 4-byte trie size header plus trie and normalized-data sections. If the blob is shorter than sizeof(uint)+1 bytes there is no usable header, so it throws this ArgumentException.","triggerScenarios":"Loading a SentencePiece normalizer whose decoded precompiled_charsmap blob is empty or only 1-4 bytes — e.g. base64 decoded to a stub, corrupted file, or wrongly extracted blob.","commonSituations":"Charsmap blob built manually from spm_normalize export with wrong byte order/truncation; test fixtures with dummy values; partial file downloads.","solutions":["Use the full precompiled_charsmap from the original SentencePiece model proto (normalizer_spec.precompiled_charsmap).","Re-export the charsmap with sentencepiece_model_pb2 rather than hand-assembling bytes.","Check the base64 string length: a valid charsmap blob is typically tens of KB."],"exampleFix":"// before: stub blob\nvar normalizer = new SentencePieceNormalizer(precompiledCharsMap: new byte[] { 1, 2, 3 }, ...);\n// after: real blob from model proto\nvar normalizer = new SentencePieceNormalizer(precompiledCharsMap: model.NormalizerSpec.PrecompiledCharsmap.ToByteArray(), ...);","handlingStrategy":"validation","validationCode":"bool CharsmapBlobPlausible(byte[] blob) => blob is { Length: > 1024 }; // real charsmaps are large; header alone needs >4 bytes","typeGuard":"bool HasValidCharsmapHeader(ReadOnlySpan<byte> blob) => blob.Length > sizeof(uint) && BitConverter.ToUInt32(blob) > 0;","tryCatchPattern":"try { var norm = new SentencePieceNormalizer(blob, ...); }\ncatch (ArgumentException ex) when (ex.Message == \"Blob for normalization rule is broken.\") {\n    // re-extract the blob from the SentencePiece model proto\n}","preventionTips":["Always source the blob from normalizer_spec.precompiled_charsmap","Never hand-truncate or dummy-fill charsmap bytes","Check blob length in your loader before constructing the normalizer"],"tags":["sentencepiece","binary-data","argument","tokenizer"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}