{"record":{"id":"31c0e1e32f03d55c","repo":"dotnet/machinelearning","slug":"malformed-varint","errorCode":null,"errorMessage":"Malformed varint.","messagePattern":"Malformed varint\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/SentencepieceModel.cs","lineNumber":63,"sourceCode":"                    return result;\n                }\n            }\n\n            // Negative int32 values are sign-extended to 10-byte varints; consume remaining bytes.\n            for (int i = 0; i < 5; i++)\n            {\n                if (pos >= end)\n                {\n                    throw new InvalidDataException(\"Unexpected end of data while reading varint.\");\n                }\n\n                if ((data[pos++] & 0x80) == 0)\n                {\n                    return result;\n                }\n            }\n\n            throw new InvalidDataException(\"Malformed varint.\");\n        }\n\n        internal static int ReadLengthPrefix(byte[] data, int end, ref int pos)\n        {\n            int length = ReadRawVarint32(data, end, ref pos);\n            if ((uint)length > (uint)(end - pos))\n            {\n                throw new InvalidDataException(\"Invalid length-delimited field size.\");\n            }\n\n            return length;\n        }\n\n        internal static string ReadString(byte[] data, int end, ref int pos)\n        {\n            int length = ReadLengthPrefix(data, end, ref pos);\n            string result = Encoding.UTF8.GetString(data, pos, length);\n            pos += length;","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/SentencepieceModel.cs#L45-L81","documentation":"Thrown at the end of ReadRawVarint32 when ten consecutive bytes all have the continuation bit set, exceeding the maximum width of a protobuf varint. This means the byte stream is not valid protobuf — the SentencePiece reader refuses to continue rather than producing an undefined value.","triggerScenarios":"ReadRawVarint32 (reached via `length`) consumes bytes at shifts 0..28 and then the negative-value loop (5 more bytes, i == 4 without terminator), totalling more than the protobuf-spec max of 10 bytes, all with bit 0x80 set. Happens when the data is not actually protobuf or is desynchronized mid-stream.","commonSituations":"Passing a non-protobuf file (e.g. a raw text vocab, a BPE JSON file, or an HTML error page saved as .model) to the SentencePiece tokenizer; parsing a stream whose field framing got desynchronized earlier; binary corruption such as wrong encoding conversion (UTF-8 re-encode of a binary file).","solutions":["Confirm the file really is a serialized SentencePiece ModelProto — open the first bytes and check for plausible protobuf framing, not text/HTML.","Re-export the model from the original source with SentencePiece/protobuf tooling instead of hand-editing or re-encoding it.","Verify the file was not transferred in text mode (FTP ASCII transfer or charset conversion corrupts binaries).","If desynchronization is suspected after an earlier error, restart parsing from offset 0 of a validated buffer rather than continuing."],"exampleFix":"// before\n// vocab.json mistakenly passed to the SentencePiece tokenizer\nvar tokenizer = SentencePieceTokenizer.Create(File.ReadAllBytes(\"vocab.json\"));\n\n// after\nif (!IsSentencePieceModel(modelPath)) // check magic/framing or correct file\n    throw new InvalidOperationException($\"'{modelPath}' is not a SentencePiece .model protobuf file.\");\nvar tokenizer = SentencePieceTokenizer.Create(File.ReadAllBytes(\"sentencepiece.bpe.model\"));","handlingStrategy":"validation","validationCode":"// Ensure the file is binary protobuf, not text/HTML/JSON:\nbool LooksBinary(byte[] d) => d.Length > 0 && (d[0] == 0x0A || char.IsControl((char)d[0]) == false && d[0] < 0x80 && d[0] != (byte)'<');\nif (!LooksBinary(modelBytes))\n    throw new InvalidDataException(\"Input does not look like a SentencePiece protobuf model.\");","typeGuard":null,"tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }\ncatch (InvalidDataException ex)\n{ throw new InvalidOperationException(\"Data is not valid SentencePiece protobuf (malformed varint). Confirm you are loading the serialized .model file, not a text vocab or error page.\", ex); }","preventionTips":["Confirm the model format before loading: .model = protobuf, vocab files are not","Check for HTML error pages saved as .model when downloads fail behind proxies","Never transfer binary models in text/ASCII mode (FTP, charset conversion)","If parse fails mid-stream, restart from offset 0 on a validated buffer"],"tags":["protobuf","varint","malformed-data","tokenizer"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}