apache/beam · error · RuntimeException

Failed to read Parquet footer for

Error message

Failed to read Parquet footer for 

What it means

CreateCDCReadTasksDoFn.getRowGroupSizes opens a Parquet data file to read its footer (column-chunk byte sizes used for split sizing). If the file cannot be read — missing, deleted, corrupt footer, or an I/O failure on the storage layer — the IOException is wrapped in a RuntimeException naming the file path.

Solutions

1. Re-run the pipeline; if it was a transient storage error, retrying with longer retention is often enough.
2. Increase VACUUM retention (delta.deletedFileRetentionDuration) and make sure VACUUM is not running concurrently with the read.
3. Verify Hadoop/storage configuration (credentials, region, endpoint) so the file is actually readable by the Beam worker.
4. Check the file exists and is readable at the printed path; if it is gone, re-list the snapshot or lower the start version to one whose files still exist.

Example fix

// before
ALTER TABLE t VACUUM RETAIN 24 HOURS; // deletes files mid-read
// after
ALTER TABLE t SET TBLPROPERTIES ('delta.deletedFileRetentionDuration'='interval 168 hours');
ALTER TABLE t VACUUM RETAIN 168 HOURS; // run outside read windows

// pre-check readability
org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.Path.getFileSystem(hadoopPath, conf);
if (!fs.exists(hadoopPath)) throw new java.io.FileNotFoundException(hadoopPath.toString());

try { /* read */ } catch (RuntimeException e) { if (e.getCause() instanceof java.io.IOException && isTransient(e.getCause())) { retryWithBackoff(); } else { throw e; } }

When it happens

Trigger: VACUUM removed the underlying data file between listing and reading; path points to a file deleted/compacted by a concurrent writer; storage outage (S3/GCS/HDFS errors); corrupt or truncated Parquet footer.

Common situations: Long-running CDC reads over a table that runs delta VACUUM on a short retention; reading a snapshot whose files were rewritten by OPTIMIZE; transient cloud-storage 5xx errors; wrong filesystem credentials configured in Hadoop conf.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to read Parquet footer for
• Beam write property ' ' is not supported. Writing to Delta…
• Cannot set both endVersion and endTimestamp.
• Cannot set both startVersion and startTimestamp.
• Cannot set both version and timestamp.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/20a0faaeef8491f8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/CreateCDCReadTasksDoFn.java:279

    for (DeltaCDCReadTask task : group) {
      out.output(task);
    }
  }

  private List<Long> getRowGroupSizes(String pathStr, Configuration conf) {
    List<Long> sizes = new ArrayList<>();
    try {
      org.apache.hadoop.fs.Path hadoopPath = new org.apache.hadoop.fs.Path(pathStr);
      org.apache.parquet.hadoop.metadata.ParquetMetadata metadata =
          org.apache.parquet.hadoop.ParquetFileReader.readFooter(
              conf,
              hadoopPath,
              org.apache.parquet.format.converter.ParquetMetadataConverter.NO_FILTER);
      for (org.apache.parquet.hadoop.metadata.BlockMetaData block : metadata.getBlocks()) {
        sizes.add(block.getTotalByteSize());
      }
    } catch (java.io.IOException e) {
      throw new RuntimeException("Failed to read Parquet footer for " + pathStr, e);
    }
    return sizes;
  }

  private static class CommitActionsInfo {
    final long version;

    final long timestamp;
    final List<Row> cdcInfo = new ArrayList<>();
    final List<Row> insertInfo = new ArrayList<>();

    CommitActionsInfo(long version, long timestamp) {
      this.version = version;
      this.timestamp = timestamp;
    }
  }
}

View on GitHub (pinned to 12126d8942)