{"record":{"id":"b46cdedc07c0121e","repo":"dotnet/machinelearning","slug":"unexpected-end-of-data-while-skipping-fixed64","errorCode":null,"errorMessage":"Unexpected end of data while skipping fixed64.","messagePattern":"Unexpected end of data while skipping fixed64\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/SentencepieceModel.cs","lineNumber":139,"sourceCode":"                            throw new InvalidDataException(\"Unexpected end of data while skipping varint.\");\n                        }\n\n                        if ((data[pos++] & 0x80) == 0)\n                        {\n                            break;\n                        }\n\n                        if (i == 9)\n                        {\n                            throw new InvalidDataException(\"Malformed varint.\");\n                        }\n                    }\n                    break;\n\n                case 1: // 64-bit fixed\n                    if (pos > end - 8)\n                    {\n                        throw new InvalidDataException(\"Unexpected end of data while skipping fixed64.\");\n                    }\n                    pos += 8;\n                    break;\n\n                case 2: // length-delimited\n                    int skipLength = ReadLengthPrefix(data, end, ref pos);\n                    pos += skipLength;\n                    break;\n\n                case 5: // 32-bit fixed\n                    if (pos > end - 4)\n                    {\n                        throw new InvalidDataException(\"Unexpected end of data while skipping fixed32.\");\n                    }\n                    pos += 4;\n                    break;\n\n                default:","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/SentencepieceModel.cs#L121-L157","documentation":"Thrown by SkipField for wire type 1 (64-bit fixed): fewer than 8 bytes remain before `end`, so the fixed64 value of an unknown field cannot be skipped. Thrown to keep the protobuf cursor consistent — the parser refuses to read past the buffer edge.","triggerScenarios":"SkipField sees an unknown field with wire type 1 and evaluates pos > end - 8. Occurs when a truncated model ends inside a fixed64 field (e.g. double-valued unknown fields added by newer protobuf definitions).","commonSituations":"Model file cut off during download inside a fixed64 field; using a model saved by a newer SentencePiece version whose extra double fields the current parser must skip; corrupted length prefixes shifting the cursor into wrong field boundaries.","solutions":["Re-download the model and verify byte-for-byte integrity (size + hash).","Use a model exported with a SentencePiece/protobuf version compatible with your library to avoid unknown fixed64 fields.","Ensure full-file reads (CopyTo loop) rather than single Read calls that may under-fill the buffer.","Validate the model with an independent protobuf decoder before passing it to the tokenizer."],"exampleFix":"// before\nvar tokenizer = SentencePieceTokenizer.Create(partialBytes); // ends mid-field\n\n// after\nbyte[] bytes = File.ReadAllBytes(modelPath);\nif (bytes.Length != expectedModelLength)\n    throw new InvalidDataException($\"Model file is {bytes.Length} bytes, expected {expectedModelLength}; re-download.\");\nvar tokenizer = SentencePieceTokenizer.Create(bytes);","handlingStrategy":"try-catch","validationCode":"if (modelBytes.Length != expectedSize)\n    throw new InvalidDataException($\"Model incomplete: {modelBytes.Length}/{expectedSize} bytes. Re-download before loading.\");","typeGuard":null,"tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }\ncatch (InvalidDataException ex)\n{ throw new InvalidDataException(\"Unknown fixed64 field truncated: the model file is incomplete. Re-download it.\", ex); }","preventionTips":["Verify download completeness against the source file size","Use models exported with a protobuf/SentencePiece version your parser understands","Copy model files with integrity-checked transfer tools (rsync -c, checksums)","Catch InvalidDataException at load time and offer automatic re-download"],"tags":["protobuf","tokenizer","fixed64","truncated-input"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}