{"record":{"id":"3c03564e55519b7e","repo":"dotnet/machinelearning","slug":"invalid-length-delimited-field-size","errorCode":null,"errorMessage":"Invalid length-delimited field size.","messagePattern":"Invalid length-delimited field size\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/SentencepieceModel.cs","lineNumber":71,"sourceCode":"                {\n                    throw new InvalidDataException(\"Unexpected end of data while reading varint.\");\n                }\n\n                if ((data[pos++] & 0x80) == 0)\n                {\n                    return result;\n                }\n            }\n\n            throw new InvalidDataException(\"Malformed varint.\");\n        }\n\n        internal static int ReadLengthPrefix(byte[] data, int end, ref int pos)\n        {\n            int length = ReadRawVarint32(data, end, ref pos);\n            if ((uint)length > (uint)(end - pos))\n            {\n                throw new InvalidDataException(\"Invalid length-delimited field size.\");\n            }\n\n            return length;\n        }\n\n        internal static string ReadString(byte[] data, int end, ref int pos)\n        {\n            int length = ReadLengthPrefix(data, end, ref pos);\n            string result = Encoding.UTF8.GetString(data, pos, length);\n            pos += length;\n            return result;\n        }\n\n        internal static float ReadFloat(byte[] data, int end, ref int pos)\n        {\n            if (pos > end - 4)\n            {\n                throw new InvalidDataException(\"Unexpected end of data while reading float.\");","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/SentencepieceModel.cs#L53-L89","documentation":"Thrown by ReadLengthPrefix when the varint length of a length-delimited field exceeds the number of bytes remaining in the buffer (length > end - pos). The library uses this check to ensure every string/bytes field (e.g. sentencepiece piece strings) lies within the buffer before reading it.","triggerScenarios":"ReadLengthPrefix (called from `length` and `skipLength`) reads a length varint whose unsigned value is larger than the remaining byte count — a field header promising more payload than the buffer contains. Caused by truncation or a corrupted length varint.","commonSituations":"Downloading a model via a connection that dropped mid-transfer; an incomplete asset embedded during build; corrupt length varint from bit flips; slicing a shared buffer with the wrong end offset so the parser sees a length prefix near the buffer edge.","solutions":["Verify and re-download the model file; compare byte length with a known-good copy.","Check the code that produces the byte slice — ensure end offsets reflect the actual buffer length.","If the model came from a package/cache, clear the cache (e.g. ~/.cache/huggingface) and let it re-fetch.","Wrap tokenizer creation in try-catch (InvalidDataException) and fail with a 're-download model' message."],"exampleFix":"// before\nvar tokenizer = SentencePieceTokenizer.Create(cachedBytes); // cache truncated\n\n// after\nbyte[] cachedBytes = LoadFromCache(modelKey);\nif (cachedBytes == null || cachedBytes.Length != knownGoodLength)\n{\n    cachedBytes = DownloadModel(modelUrl);\n    SaveToCache(modelKey, cachedBytes);\n}\nvar tokenizer = SentencePieceTokenizer.Create(cachedBytes);","handlingStrategy":"validation","validationCode":"if (modelBytes == null || modelBytes.Length != expectedModelLength)\n    throw new InvalidDataException($\"Model size mismatch: got {modelBytes?.Length ?? 0} bytes, expected {expectedModelLength}. Re-download.\");","typeGuard":null,"tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }\ncatch (InvalidDataException ex)\n{ throw new InvalidDataException(\"Length-delimited field overruns buffer — model is truncated. Re-download the tokenizer model.\", ex); }","preventionTips":["Store expected model length/hash alongside the model path or cache key","Clear and rebuild the model cache after failed downloads","Audit code that slices buffers — ensure end offsets equal data.Length","Use resumable, verified downloads (Content-Length check) for large models"],"tags":["protobuf","tokenizer","truncated-input","length-prefix"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}