apache/beam · error · RuntimeException

Failed to read Parquet footer for

Error message

Failed to read Parquet footer for 

What it means

Same family as the CDC variant: CreateReadTasksDoFn.getRowGroupSizes reads the Parquet footer of a Delta data file to compute row-group byte sizes for task splitting, and wraps any IOException reading that footer in a RuntimeException naming the file.

Solutions

1. Retry the read; transient object-store errors resolve on re-run.
2. Ensure the requested version's files still exist — avoid reading versions older than delta.deletedFileRetentionDuration.
3. Fix storage configuration on the worker (credentials, region, endpoint) so the path is readable.
4. Inspect the printed path with a standalone reader (e.g. parquet-tools) to confirm corruption vs. access problems.

Example fix

// before
DeltaIO.read().from("s3://bucket/t").withVersion(1); // version 1 files vacuumed
// after
DeltaIO.read().from("s3://bucket/t").withVersion(latestAvailableNonVacuumedVersion);

// pre-check readability
org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.Path.getFileSystem(hadoopPath, conf);
if (!fs.exists(hadoopPath)) throw new java.io.FileNotFoundException(hadoopPath.toString());

try { /* read */ } catch (RuntimeException e) { if (e.getCause() instanceof java.io.IOException && isTransient(e.getCause())) { retryWithBackoff(); } else { throw e; } }

When it happens

Trigger: Snapshot data file deleted by VACUUM or rewritten by a concurrent writer before read; missing/wrong storage credentials in the Hadoop Configuration; truncated or corrupt Parquet file; transient S3/GCS/HDFS I/O error.

Common situations: Reading an old version/timestamp of a Delta table whose files have since been vacuumed; network blips to object storage; IAM roles lacking GetObject on the table location.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to read Parquet footer for
• Beam write property ' ' is not supported. Writing to Delta…
• Cannot set both endVersion and endTimestamp.
• Cannot set both startVersion and startTimestamp.
• Cannot set both version and timestamp.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b0f2f5b92226699d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/CreateReadTasksDoFn.java:157

      out.output(readTask);
    }
  }

  private List<Long> getRowGroupSizes(SerializableRow scanFileRow, Configuration conf) {
    List<Long> sizes = new ArrayList<>();
    String pathStr = InternalScanFileUtils.getAddFileStatus(scanFileRow).getPath();
    try {
      org.apache.hadoop.fs.Path hadoopPath = new org.apache.hadoop.fs.Path(pathStr);
      org.apache.parquet.hadoop.metadata.ParquetMetadata metadata =
          org.apache.parquet.hadoop.ParquetFileReader.readFooter(
              conf,
              hadoopPath,
              org.apache.parquet.format.converter.ParquetMetadataConverter.NO_FILTER);
      for (org.apache.parquet.hadoop.metadata.BlockMetaData block : metadata.getBlocks()) {
        sizes.add(block.getTotalByteSize());
      }
    } catch (java.io.IOException e) {
      throw new RuntimeException("Failed to read Parquet footer for " + pathStr, e);
    }
    return sizes;
  }
}

View on GitHub (pinned to 12126d8942)