{"record":{"id":"3e771987ea9c646a","repo":"dotnet/machinelearning","slug":"unexpected-end-of-data-while-reading-float","errorCode":null,"errorMessage":"Unexpected end of data while reading float.","messagePattern":"Unexpected end of data while reading float\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/SentencepieceModel.cs","lineNumber":89,"sourceCode":"                throw new InvalidDataException(\"Invalid length-delimited field size.\");\n            }\n\n            return length;\n        }\n\n        internal static string ReadString(byte[] data, int end, ref int pos)\n        {\n            int length = ReadLengthPrefix(data, end, ref pos);\n            string result = Encoding.UTF8.GetString(data, pos, length);\n            pos += length;\n            return result;\n        }\n\n        internal static float ReadFloat(byte[] data, int end, ref int pos)\n        {\n            if (pos > end - 4)\n            {\n                throw new InvalidDataException(\"Unexpected end of data while reading float.\");\n            }\n\n            float value;\n            if (BitConverter.IsLittleEndian)\n            {\n                value = BitConverter.ToSingle(data, pos);\n            }\n            else\n            {\n                // Protobuf fixed32 is always little-endian; reverse bytes on big-endian platforms.\n                byte[] buffer = new byte[4];\n                buffer[0] = data[pos + 3];\n                buffer[1] = data[pos + 2];\n                buffer[2] = data[pos + 1];\n                buffer[3] = data[pos];\n                value = BitConverter.ToSingle(buffer, 0);\n            }\n","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/SentencepieceModel.cs#L71-L107","documentation":"Thrown by SentencePieceProtobufReader.ReadFloat when fewer than 4 bytes remain before `end`, so a 32-bit IEEE float field (e.g. sentencepiece piece `score`) cannot be read. The library checks explicitly to avoid reading past the buffer and to report the stream as truncated.","triggerScenarios":"ReadFloat encounters pos > end - 4 — a float-typed field header was parsed but the remaining payload holds fewer than 4 bytes. Occurs when truncation lands inside a repeated score/array field of the ModelProto.","commonSituations":"Model file truncated inside a large repeated float array (common in sentencepiece models with thousands of scores); chunked HTTP download assembled with a dropped chunk; memory-mapped or span-based reads with a wrong window end.","solutions":["Re-download the model file and verify its checksum — truncation inside a float array is the dominant cause.","Reassemble streamed downloads correctly (use CopyTo/ReadLoop, not single Read calls) before parsing.","Validate the model with the official protobuf deserializer as a pre-check in your pipeline.","Fail fast at startup with checksum verification rather than deep inside tokenization."],"exampleFix":"// before\nbyte[] buf = new byte[fileSize];\nusing var fs = File.OpenRead(path);\nint read = fs.Read(buf, 0, buf.Length); // may read fewer bytes\nvar tokenizer = SentencePieceTokenizer.Create(buf);\n\n// after\nbyte[] buf;\nusing (var ms = new MemoryStream())\n{\n    using var fs = File.OpenRead(path);\n    fs.CopyTo(ms);\n    buf = ms.ToArray();\n}\nvar tokenizer = SentencePieceTokenizer.Create(buf);","handlingStrategy":"validation","validationCode":"// Ensure the whole file was read before parsing:\nbyte[] ReadAllBytes(string path)\n{\n    using var fs = File.OpenRead(path);\n    using var ms = new MemoryStream();\n    fs.CopyTo(ms);\n    var bytes = ms.ToArray();\n    if (bytes.Length != new FileInfo(path).Length)\n        throw new InvalidDataException(\"Incomplete read of tokenizer model.\");\n    return bytes;\n}","typeGuard":null,"tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelBytes); }\ncatch (InvalidDataException ex)\n{ throw new InvalidDataException(\"Model stream truncated inside a float field — re-download and verify checksum.\", ex); }","preventionTips":["Use CopyTo loops, never assume a single Read fills the buffer","Verify downloads against Content-Length and checksum before parsing","When memory-mapping/splicing buffers, double-check the end offset covers the full model","Load models once at startup, inside a try-catch with a re-download fallback"],"tags":["protobuf","tokenizer","float","truncated-input"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}