apache/iceberg · error · IOException

Incorrect length provided for file

Error message

Incorrect length provided for file <location>, given a length of <length> and did not reach the end of stream

What it means

EagerInputFile.newStream reads the whole file into memory using the length supplied at construction. If after reading `length` bytes the stream is not yet at EOF, the actual file is longer than the declared length, so the eager read would silently truncate. It throws IOException (later wrapped as RuntimeIOException) to prevent reading partial data.

Solutions

1. Construct the eager file with the actual current file size (file.length() / accurate manifest length).
2. Drop the explicit length and let the InputFile determine it from storage.
3. Recompute table/manifest metadata if file sizes are stale.
4. Verify the file was not modified after the length was captured (Iceberg files must be immutable).

Example fix

// before
InputFile f = InputFile.local(path, 1024); // hardcoded length
// after
long len = Files.size(Paths.get(path));
InputFile f = InputFile.local(path, len);

long real = file.getLength();
Preconditions.checkArgument(declaredLength == real,
    "Declared length %s != actual %s", declaredLength, real);

try {
  return eagerFile.newStream();
} catch (RuntimeIOException e) {
  if (e.getMessage().contains("Incorrect length")) {
    return io.newInputFile(location).newStream(); // lazy, derives length itself
  }
  throw e;
}

When it happens

Trigger: Creating EagerInputFile (e.g. InputFile.local(?) or fromCache with an explicit length) where the length parameter is smaller than the real file size, then calling newStream().

Common situations: Passing a stale/incorrect length taken from old metadata or manifest stats; hardcoding a length; a file that grew after the length was captured; manifest entries with wrong file size.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• An error occurred while aborting the stream
• An error occurred while closing the stream
• Bulk deletion failed
• Can't create file
• Can't get Stripe's length from the file writer with path

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/9a5d2530f53c0f5e. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/io/EagerInputFile.java:80

  @Override
  public String location() {
    return delegate.location();
  }

  @Override
  public boolean exists() {
    return delegate.exists();
  }

  @Override
  public SeekableInputStream newStream() {
    byte[] bytes = new byte[(int) length];
    try (SeekableInputStream src = delegate.newStream()) {
      IOUtil.readFully(src, bytes, 0, bytes.length);
      // reads from the already open stream; no additional request
      if (src.read() != -1) {
        throw new IOException(
            "Incorrect length provided for file "
                + delegate.location()
                + ", given a length of "
                + length
                + " and did not reach the end of stream");
      }
    } catch (IOException e) {
      throw new RuntimeIOException(e, "Failed to fetch file: %s", delegate.location());
    }
    return new EagerInputStream(bytes);
  }

  /** An {@link EagerInputFile} that preserves the delegate's Hadoop configuration. */
  private static class EagerInputFileConfigurable extends EagerInputFile
      implements HadoopConfigurable {

    private final HadoopConfigurable delegate;

View on GitHub (pinned to 86d9c8fc54)