{"record":{"id":"146dc298c722fa62","repo":"dotnet/machinelearning","slug":"unexpected-end-of-data-while-skipping-varint","errorCode":null,"errorMessage":"Unexpected end of data while skipping varint.","messagePattern":"Unexpected end of data while skipping varint\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/SentencepieceModel.cs","lineNumber":121,"sourceCode":"                buffer[2] = data[pos + 1];\n                buffer[3] = data[pos];\n                value = BitConverter.ToSingle(buffer, 0);\n            }\n\n            pos += 4;\n            return value;\n        }\n\n        internal static void SkipField(byte[] data, int end, int wireType, ref int pos)\n        {\n            switch (wireType)\n            {\n                case 0: // varint (max 10 bytes per protobuf spec)\n                    for (int i = 0; i < 10; i++)\n                    {\n                        if (pos >= end)\n                        {\n                            throw new InvalidDataException(\"Unexpected end of data while skipping varint.\");\n                        }\n\n                        if ((data[pos++] & 0x80) == 0)\n                        {\n                            break;\n                        }\n\n                        if (i == 9)\n                        {\n                            throw new InvalidDataException(\"Malformed varint.\");\n                        }\n                    }\n                    break;\n\n                case 1: // 64-bit fixed\n                    if (pos > end - 8)\n                    {\n                        throw new InvalidDataException(\"Unexpected end of data while skipping fixed64.\");","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/SentencepieceModel.cs#L103-L139","documentation":"Thrown by SkipField when handling wire type 0 (varint): a continuation byte is required (high bit set) but pos >= end, so the unknown field's varint cannot be fully skipped. The library must consume the whole value to stay synchronized with the protobuf stream; it throws when the data ends too early.","triggerScenarios":"SkipField encounters an unknown field with wire type 0, iterates up to 10 varint bytes, and hits the end of the buffer before a byte with the continuation bit clear. Triggered by truncated models containing fields (e.g. newer ModelProto fields like trainer_spec variants) that must be skipped.","commonSituations":"Model serialized by a newer SentencePiece/protobuf version includes unknown fields; the (truncated) file ends inside such a field; partial downloads landing mid-unknown-field; binary corruption extending a varint past the buffer.","solutions":["Re-download/verify the complete model file — the parse cannot recover from a mid-field truncation.","Regenerate the model with a compatible SentencePiece version if unknown-field handling matters to you.","Ensure the whole file is read into memory before parsing (no partial reads).","Add a checksum check before tokenizer construction to detect truncation early."],"exampleFix":"// before\nvar tokenizer = SentencePieceTokenizer.Create(truncatedStreamBytes);\n\n// after\nbyte[] bytes = ReadAllBytesFully(stream); // loop until EOF\nif (!ChecksumMatches(bytes))\n    throw new InvalidDataException(\"Tokenizer model incomplete; re-download.\");\nvar tokenizer = SentencePieceTokenizer.Create(bytes);","handlingStrategy":"try-catch","validationCode":"// Detect truncation cheaply before loading:\nif (modelBytes.Length == 0 || modelBytes[^1] == 0x80) // ends mid-continuation byte\n    throw new InvalidDataException(\"Model appears truncated (ends on a varint continuation byte).\");","typeGuard":null,"tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }\ncatch (InvalidDataException ex)\n{ throw new InvalidDataException(\"Could not skip an unknown field: model stream is truncated or from an incompatible protobuf version.\", ex); }","preventionTips":["Regenerate models with a SentencePiece version compatible with your library","Verify checksums before construction so truncation is caught early","Read complete files; avoid partial reads from network streams","Log the model file path and size in the failure message to speed diagnosis"],"tags":["protobuf","tokenizer","skip-field","truncated-input"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}