{"record":{"id":"35190ecd9b5a2d1c","repo":"dotnet/machinelearning","slug":"the-number-of-bytes-read-from-stream-is-less-than","errorCode":null,"errorMessage":"The number of bytes read from stream is less than expected. Please check the data files.","messagePattern":"The number of bytes read from stream is less than expected\\. Please check the data files\\.","errorType":"exception","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.TorchSharp/Utils/FileUtils.cs","lineNumber":58,"sourceCode":"        /// <exception cref=\"ArgumentException\"/>\n        /// <exception cref=\"InvalidDataException\">When the contents in the stream don't match the need.</exception>\n        public static IEnumerable<T> LoadNumberArrayFromStream<T>(Stream stream, int numElements, int tSize)\n        {\n            if (stream == null || !stream.CanRead)\n            {\n                throw new ArgumentException($\"Stream should be non-null and its stream.CanRead property should be true.\");\n            }\n            if (!_validTypes.Contains(typeof(T)))\n            {\n                throw new NotSupportedException($\"Type {typeof(T)} not supported in data loading.\");\n            }\n\n            var numBytesConsumed = numElements * tSize;\n            var byteBuffer = new byte[numBytesConsumed];\n            var numBytesRead = stream.Read(byteBuffer, 0, numBytesConsumed);\n            if (numBytesConsumed != numBytesRead)\n            {\n                throw new InvalidDataException(\n                    $\"The number of bytes read from stream is less than expected. Please check the data files.\");\n            }\n\n            var targetBuffer = new T[numBytesConsumed / tSize];\n            Buffer.BlockCopy(byteBuffer, 0, targetBuffer, 0, numBytesConsumed);\n            return targetBuffer;\n        }\n    }\n}\n","sourceCodeStart":40,"sourceCodeEnd":68,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.TorchSharp/Utils/FileUtils.cs#L40-L68","documentation":"After reading numElements * tSize bytes from the stream, LoadNumberArrayFromStream verifies that the number of bytes read matches what was requested and throws InvalidDataException on a shortfall. A short read means the underlying data file is truncated, corrupted, or numElements/tSize don't describe it correctly.","triggerScenarios":"Calling LoadNumberArrayFromStream with numElements or tSize larger than the data actually present in the stream — e.g., reading with numElements=10000, tSize=4 from a truncated file, a partially downloaded checkpoint, or a stream whose position was already advanced past most of the data.","commonSituations":"Incomplete model download (network interruption), wrong numElements constant for the checkpoint format, mixing up tSize (reading 8-byte doubles from 4-byte float data overruns the file), or reading a stream that a previous reader already consumed.","solutions":["Verify the data file is complete and uncorrupted (re-download or re-export the model checkpoint).","Check that numElements and tSize match the actual format: `long expected = (long)numElements * tSize; if (stream.Length - stream.Position < expected) fail;`.","Reset stream.Position to 0 before loading if the stream was previously read.","Wrap the call in try-catch on InvalidDataException and prompt the user to re-acquire the model files."],"exampleFix":"// before\nvar w = FileUtils.LoadNumberArrayFromStream<float>(stream, 30522 * 768, 4); // truncated file: throws\n\n// after\nif (stream.Length - stream.Position < 30522L * 768 * 4)\n    throw new InvalidDataException(\"Model file incomplete; re-download it.\");\nvar w = FileUtils.LoadNumberArrayFromStream<float>(stream, 30522 * 768, 4);","handlingStrategy":"try-catch","validationCode":"long expected = (long)numElements * tSize;\nif (stream.Length - stream.Position < expected)\n    throw new InvalidDataException(\"Stream too short for the requested data; file may be truncated.\");","typeGuard":null,"tryCatchPattern":"try { var data = FileUtils.LoadNumberArrayFromStream<float>(stream, n, 4); }\ncatch (InvalidDataException ex) { log.LogError(ex, \"Model data truncated or wrong format; re-download the checkpoint.\"); throw new CorruptModelException(...); }","preventionTips":["Verify checkpoint file integrity (size/hash) after download.","Set stream.Position = 0 before loading a previously-read stream.","Match numElements and tSize to the checkpoint's documented format.","Avoid shared stream readers that advance the position unexpectedly."],"tags":["csharp","stream","invalid-data","file-corruption"],"backgroundTag":"file-read-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}