{"record":{"id":"48e6a6d87d095242","repo":"dotnet/machinelearning","slug":"unexpected-end-of-data-while-reading-varint","errorCode":null,"errorMessage":"Unexpected end of data while reading varint.","messagePattern":"Unexpected end of data while reading varint\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/SentencepieceModel.cs","lineNumber":24,"sourceCode":"// Replaces a full Google.Protobuf dependency with just enough wire-format reading\n// to parse the fields the tokenizer implementation actually consumes.\n// SentencePiece is under the Apache License 2.0 https://github.com/google/sentencepiece/blob/master/LICENSE\n\nusing System;\nusing System.Collections.Generic;\nusing System.IO;\nusing System.Text;\n\nnamespace Sentencepiece\n{\n    /// <summary>Low-level protobuf wire-format primitives (read-only, forward-only).</summary>\n    internal static class SentencePieceProtobufReader\n    {\n        internal static int ReadRawVarint32(byte[] data, int end, ref int pos)\n        {\n            if (pos >= end)\n            {\n                throw new InvalidDataException(\"Unexpected end of data while reading varint.\");\n            }\n\n            byte b = data[pos++];\n            int result = b & 0x7F;\n            if ((b & 0x80) == 0)\n            {\n                return result;\n            }\n\n            for (int shift = 7; shift < 32; shift += 7)\n            {\n                if (pos >= end)\n                {\n                    throw new InvalidDataException(\"Unexpected end of data while reading varint.\");\n                }\n\n                b = data[pos++];\n                result |= (b & 0x7F) << shift;","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/SentencepieceModel.cs#L6-L42","documentation":"This InvalidDataException is thrown by SentencePieceProtobufReader.ReadRawVarint32 when the reader position is already at or past the end of the data buffer before the first varint byte can be read. The library throws it while parsing a SentencePiece tokenizer model file (Protobuf wire format) to signal the byte stream is truncated or the framing is wrong, rather than returning garbage.","triggerScenarios":"Calling ReadRawVarint32 (e.g. via ReadLengthPrefix/`length`) when `pos >= end`, i.e. the varint field begins exactly at the end of the buffer. Produced when a field header claims a field exists but there are zero bytes remaining — typically a truncated model file or a length prefix that overruns the buffer.","commonSituations":"Loading a truncated sentencepiece.bpe.model (incomplete download or HTTP stream cut off); a model file that was corrupted or partially written; passing a byte array containing only part of the serialized ModelProto to the tokenizer constructor; embedding a model resource that was mis-sized during build/packaging.","solutions":["Re-download or re-copy the tokenizer model file and verify its byte size/checksum against the source.","Check that the full serialized SentencePiece model (e.g. sentencepiece.bpe.model) is being passed, not a fragment — log data.Length before parsing.","If the model is embedded as a resource, rebuild/re-embed it and confirm the embedded resource length matches the original file.","Wrap model loading in a try-catch for InvalidDataException and surface a clear 'corrupt or truncated tokenizer model' message to the user."],"exampleFix":"// before\nbyte[] modelData = File.ReadAllBytes(partialDownloadPath);\nvar tokenizer = SentencePieceTokenizer.Create(modelData);\n\n// after\nbyte[] modelData = File.ReadAllBytes(modelPath);\nif (modelData.Length == 0 || !VerifyChecksum(modelData, expectedSha256))\n{\n    throw new InvalidDataException($\"SentencePiece model at '{modelPath}' is truncated or corrupt; re-download it.\");\n}\nvar tokenizer = SentencePieceTokenizer.Create(modelData);","handlingStrategy":"validation","validationCode":"if (modelBytes == null || modelBytes.Length == 0)\n    throw new InvalidDataException(\"SentencePiece model is empty; re-download it.\");\n// checksum check where a known hash exists:\nusing var sha = System.Security.Cryptography.SHA256.Create();\nstring hash = Convert.ToHexString(sha.ComputeHash(modelBytes));\nif (hash != expectedHash)\n    throw new InvalidDataException(\"SentencePiece model failed integrity check (truncated or corrupt).\");","typeGuard":null,"tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }\ncatch (InvalidDataException ex)\n{ throw new ApplicationException(\"Tokenizer model is truncated or corrupt; re-download the .model file.\", ex); }","preventionTips":["Verify model file size/checksum before first use","Read the entire file into memory (CopyTo loop) — never single Read calls","Pin model downloads to trusted URLs and validate after download","If embedding the model, verify the embedded resource length at build time"],"tags":["protobuf","tokenizer","invalid-data","truncated-input"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}