{"record":{"id":"6156aa5f0eb77c98","repo":"dotnet/machinelearning","slug":"unexpected-end-of-data-while-skipping-fixed32","errorCode":null,"errorMessage":"Unexpected end of data while skipping fixed32.","messagePattern":"Unexpected end of data while skipping fixed32\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/SentencepieceModel.cs","lineNumber":152,"sourceCode":"                    break;\n\n                case 1: // 64-bit fixed\n                    if (pos > end - 8)\n                    {\n                        throw new InvalidDataException(\"Unexpected end of data while skipping fixed64.\");\n                    }\n                    pos += 8;\n                    break;\n\n                case 2: // length-delimited\n                    int skipLength = ReadLengthPrefix(data, end, ref pos);\n                    pos += skipLength;\n                    break;\n\n                case 5: // 32-bit fixed\n                    if (pos > end - 4)\n                    {\n                        throw new InvalidDataException(\"Unexpected end of data while skipping fixed32.\");\n                    }\n                    pos += 4;\n                    break;\n\n                default:\n                    throw new InvalidDataException($\"Unknown or unsupported protobuf wire type {wireType}.\");\n            }\n        }\n    }\n\n    /// <summary>Lightweight replacement for Google.Protobuf.ByteString with a Span property.</summary>\n    internal readonly struct SentencePieceByteString(byte[] data, int offset, int length)\n    {\n        internal ReadOnlySpan<byte> Span => data is null ? ReadOnlySpan<byte>.Empty : data.AsSpan(offset, length);\n    }\n\n    /// <summary>ModelProto  (top-level message; field numbers match sentencepiece_model.proto)</summary>\n    internal sealed class ModelProto","sourceCodeStart":134,"sourceCodeEnd":170,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/SentencepieceModel.cs#L134-L170","documentation":"Thrown by SkipField for wire type 5 (32-bit fixed): fewer than 4 bytes remain before `end`, so the fixed32 value of an unknown field cannot be skipped. Like the other SkipField checks, it prevents out-of-bounds reads and flags the SentencePiece model stream as truncated.","triggerScenarios":"SkipField encounters an unknown field with wire type 5 and evaluates pos > end - 4. Occurs when a truncated or corrupted model ends inside (or right at the start of) a fixed32 field, or after cursor desynchronization misreads a header as wire type 5.","commonSituations":"Download interrupted within the last bytes of the model (fixed32 fields such as float scores' unknown counterparts often sit late in the file); files copied incompletely (e.g. interrupted scp/rsync); damaged cache files.","solutions":["Re-download/re-copy the model file and verify checksum; incomplete transfers are the usual cause.","Clear any model cache and re-fetch (HuggingFace cache, app-local cache, etc.).","Confirm the file is a raw SentencePiece ModelProto, not compressed or wrapped (tar/zip member extracted incorrectly).","Wrap tokenizer creation in try-catch (InvalidDataException) with a retry that re-downloads the model."],"exampleFix":"// before\ntry { tokenizer = SentencePieceTokenizer.Create(cacheBytes); }\ncatch (InvalidDataException) { throw; } // dead end\n\n// after\ntry\n{\n    tokenizer = SentencePieceTokenizer.Create(cacheBytes);\n}\ncatch (InvalidDataException ex)\n{\n    Log.Warn(\"Tokenizer model corrupt, re-downloading\", ex);\n    cacheBytes = DownloadModel(modelUrl);\n    SaveToCache(modelKey, cacheBytes);\n    tokenizer = SentencePieceTokenizer.Create(cacheBytes);\n}","handlingStrategy":"try-catch","validationCode":"bool IsCompleteDownload(byte[] bytes, long expectedLength) => bytes != null && bytes.LongLength == expectedLength;\nif (!IsCompleteDownload(modelBytes, expectedLength))\n    throw new InvalidDataException(\"Incomplete model download; refusing to parse.\");","typeGuard":null,"tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }\ncatch (InvalidDataException ex)\n{\n    PurgeCache(modelKey);\n    modelBytes = DownloadModel(modelUrl);\n    tokenizer = SentencePieceTokenizer.Create(modelBytes);\n}","preventionTips":["Re-copy the model with checksum verification after any transfer failure","Clear corrupted caches (HuggingFace/app cache) and re-fetch","Check the tail of the file — fixed32 fields near EOF are hit first by truncation","Keep a fallback: on InvalidDataException, re-download automatically before surfacing the error"],"tags":["protobuf","tokenizer","fixed32","truncated-input"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}