dotnet/machinelearning · error · InvalidDataException

The number of bytes read from stream is less than expected…

Error message

The number of bytes read from stream is less than expected. Please check the data files.

What it means

After reading numElements * tSize bytes from the stream, LoadNumberArrayFromStream verifies that the number of bytes read matches what was requested and throws InvalidDataException on a shortfall. A short read means the underlying data file is truncated, corrupted, or numElements/tSize don't describe it correctly.

Solutions

1. Verify the data file is complete and uncorrupted (re-download or re-export the model checkpoint).
2. Check that numElements and tSize match the actual format: `long expected = (long)numElements * tSize; if (stream.Length - stream.Position < expected) fail;`.
3. Reset stream.Position to 0 before loading if the stream was previously read.
4. Wrap the call in try-catch on InvalidDataException and prompt the user to re-acquire the model files.

Example fix

// before
var w = FileUtils.LoadNumberArrayFromStream<float>(stream, 30522 * 768, 4); // truncated file: throws

// after
if (stream.Length - stream.Position < 30522L * 768 * 4)
    throw new InvalidDataException("Model file incomplete; re-download it.");
var w = FileUtils.LoadNumberArrayFromStream<float>(stream, 30522 * 768, 4);

long expected = (long)numElements * tSize;
if (stream.Length - stream.Position < expected)
    throw new InvalidDataException("Stream too short for the requested data; file may be truncated.");

try { var data = FileUtils.LoadNumberArrayFromStream<float>(stream, n, 4); }
catch (InvalidDataException ex) { log.LogError(ex, "Model data truncated or wrong format; re-download the checkpoint."); throw new CorruptModelException(...); }

When it happens

Trigger: Calling LoadNumberArrayFromStream with numElements or tSize larger than the data actually present in the stream — e.g., reading with numElements=10000, tSize=4 from a truncated file, a partially downloaded checkpoint, or a stream whose position was already advanced past most of the data.

Common situations: Incomplete model download (network interruption), wrong numElements constant for the checkpoint format, mixing up tSize (reading 8-byte doubles from 4-byte float data overruns the file), or reading a stream that a previous reader already consumed.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Stream should be non-null and its stream.CanRead property…
• throw new ArgumentNullException(nameof(mergeStream));
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabularyStream));
• A PrimitiveDataViewType cannot have a disposable RawType

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/35190ecd9b5a2d1c. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.TorchSharp/Utils/FileUtils.cs:58

        /// <exception cref="ArgumentException"/>
        /// <exception cref="InvalidDataException">When the contents in the stream don't match the need.</exception>
        public static IEnumerable<T> LoadNumberArrayFromStream<T>(Stream stream, int numElements, int tSize)
        {
            if (stream == null || !stream.CanRead)
            {
                throw new ArgumentException($"Stream should be non-null and its stream.CanRead property should be true.");
            }
            if (!_validTypes.Contains(typeof(T)))
            {
                throw new NotSupportedException($"Type {typeof(T)} not supported in data loading.");
            }

            var numBytesConsumed = numElements * tSize;
            var byteBuffer = new byte[numBytesConsumed];
            var numBytesRead = stream.Read(byteBuffer, 0, numBytesConsumed);
            if (numBytesConsumed != numBytesRead)
            {
                throw new InvalidDataException(
                    $"The number of bytes read from stream is less than expected. Please check the data files.");
            }

            var targetBuffer = new T[numBytesConsumed / tSize];
            Buffer.BlockCopy(byteBuffer, 0, targetBuffer, 0, numBytesConsumed);
            return targetBuffer;
        }
    }
}

View on GitHub (pinned to 7b76e69cf9)