dotnet/machinelearning · error · IOException

File {path} too big to open.

Error message

File {path} too big to open.

What it means

When loading an image file into the buffer, ImageLoader opens the file with a FileStream and checks its length. Files larger than int.MaxValue bytes (~2 GB) cannot be represented in the int-sized byte buffer, so an IOException is thrown. Note a zero-length reported size is tolerated by re-reading with File.ReadAllBytes because some filesystems (e.g. Linux procfs) report 0 for non-empty files.

Source

Thrown at src/Microsoft.ML.ImageAnalytics/ImageLoader.cs:306

                            var editor = VBufferEditor.Create(ref dst, 0);
                            dst = editor.Commit();
                        }

                    };

                return del;
            }

            private static bool TryLoadDataIntoBuffer(string path, ref VBuffer<byte> imgData)
            {
                int count = -1;
                int bytesread = -1;
                // bufferSize == 1 used to avoid unnecessary buffer in FileStream
                using (FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize: 1))
                {
                    long fileLength = fs.Length;
                    if (fileLength > int.MaxValue)
                        throw new IOException($"File {path} too big to open.");
                    else if (fileLength == 0)
                    {
                        byte[] imageBuffer;
                        // Some file systems (e.g. procfs on Linux) return 0 for length even when there's content.
                        // Thus we need to assume 0 doesn't mean empty.
                        imageBuffer = File.ReadAllBytes(path);
                        count = imageBuffer.Length;
                        imgData = new VBuffer<byte>(count, imageBuffer);
                        return (count > 0);
                    }

                    count = (int)fileLength;
                    var editor = VBufferEditor.Create(ref imgData, count);
                    bytesread = ReadToEnd(fs, editor.Values);
                    imgData = editor.Commit();
                    return (count > 0);
                }

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Remove or downsample the oversized file so each image is well under 2 GB
  2. Pre-filter input rows/files by new FileInfo(path).Length <= int.MaxValue before feeding the loader
  3. If huge images are genuinely required, read them via a library/streaming path supporting long-length data instead of ML.NET ImageLoader

Example fix

// before
loader.LoadImage(row.Path); // throws if > 2GB
// after
var fi = new FileInfo(row.Path);
if (fi.Length > int.MaxValue)
    throw new InvalidOperationException($"Image {row.Path} exceeds 2GB limit; resize or split it.");
loader.LoadImage(row.Path);
Defensive patterns

Strategy: validation

Validate before calling

var fi = new FileInfo(path);
if (fi.Length > int.MaxValue)
    throw new InvalidOperationException($"{path} exceeds 2GB image limit.");

Type guard

bool IsLoadableImageFile(string path) => File.Exists(path) && new FileInfo(path).Length <= int.MaxValue;

Try / catch

try
{
    LoadImage(path);
}
catch (IOException ex) when (ex.Message.Contains("too big to open"))
{
    logger.LogWarning("Skipping oversized image {Path}", path);
}

Prevention

When it happens

Trigger: Calling an ImageLoader/ImageLoadingTransformer load path (via TryLoadDataIntoBuffer, reached when the loader processes a row) on an image file whose size exceeds 2,147,483,647 bytes.

Common situations: Pointing an image loader at raw video frames, medical/scientific imagery, TIFF stacks, or accidentally at non-image large binaries (database dumps, archives) sitting in the watched folder.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/94928184a27f3397. Report an issue: GitHub.