{"record":{"id":"20a0faaeef8491f8","repo":"apache/beam","slug":"failed-to-read-parquet-footer-for","errorCode":null,"errorMessage":"Failed to read Parquet footer for ","messagePattern":"Failed to read Parquet footer for ","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/CreateCDCReadTasksDoFn.java","lineNumber":279,"sourceCode":"    for (DeltaCDCReadTask task : group) {\n      out.output(task);\n    }\n  }\n\n  private List<Long> getRowGroupSizes(String pathStr, Configuration conf) {\n    List<Long> sizes = new ArrayList<>();\n    try {\n      org.apache.hadoop.fs.Path hadoopPath = new org.apache.hadoop.fs.Path(pathStr);\n      org.apache.parquet.hadoop.metadata.ParquetMetadata metadata =\n          org.apache.parquet.hadoop.ParquetFileReader.readFooter(\n              conf,\n              hadoopPath,\n              org.apache.parquet.format.converter.ParquetMetadataConverter.NO_FILTER);\n      for (org.apache.parquet.hadoop.metadata.BlockMetaData block : metadata.getBlocks()) {\n        sizes.add(block.getTotalByteSize());\n      }\n    } catch (java.io.IOException e) {\n      throw new RuntimeException(\"Failed to read Parquet footer for \" + pathStr, e);\n    }\n    return sizes;\n  }\n\n  private static class CommitActionsInfo {\n    final long version;\n\n    final long timestamp;\n    final List<Row> cdcInfo = new ArrayList<>();\n    final List<Row> insertInfo = new ArrayList<>();\n\n    CommitActionsInfo(long version, long timestamp) {\n      this.version = version;\n      this.timestamp = timestamp;\n    }\n  }\n}\n","sourceCodeStart":261,"sourceCodeEnd":297,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/CreateCDCReadTasksDoFn.java#L261-L297","documentation":"CreateCDCReadTasksDoFn.getRowGroupSizes opens a Parquet data file to read its footer (column-chunk byte sizes used for split sizing). If the file cannot be read — missing, deleted, corrupt footer, or an I/O failure on the storage layer — the IOException is wrapped in a RuntimeException naming the file path.","triggerScenarios":"VACUUM removed the underlying data file between listing and reading; path points to a file deleted/compacted by a concurrent writer; storage outage (S3/GCS/HDFS errors); corrupt or truncated Parquet footer.","commonSituations":"Long-running CDC reads over a table that runs delta VACUUM on a short retention; reading a snapshot whose files were rewritten by OPTIMIZE; transient cloud-storage 5xx errors; wrong filesystem credentials configured in Hadoop conf.","solutions":["Re-run the pipeline; if it was a transient storage error, retrying with longer retention is often enough.","Increase VACUUM retention (delta.deletedFileRetentionDuration) and make sure VACUUM is not running concurrently with the read.","Verify Hadoop/storage configuration (credentials, region, endpoint) so the file is actually readable by the Beam worker.","Check the file exists and is readable at the printed path; if it is gone, re-list the snapshot or lower the start version to one whose files still exist."],"exampleFix":"// before\nALTER TABLE t VACUUM RETAIN 24 HOURS; // deletes files mid-read\n// after\nALTER TABLE t SET TBLPROPERTIES ('delta.deletedFileRetentionDuration'='interval 168 hours');\nALTER TABLE t VACUUM RETAIN 168 HOURS; // run outside read windows","handlingStrategy":"retry","validationCode":"// pre-check readability\norg.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.Path.getFileSystem(hadoopPath, conf);\nif (!fs.exists(hadoopPath)) throw new java.io.FileNotFoundException(hadoopPath.toString());","typeGuard":null,"tryCatchPattern":"try { /* read */ } catch (RuntimeException e) { if (e.getCause() instanceof java.io.IOException && isTransient(e.getCause())) { retryWithBackoff(); } else { throw e; } }","preventionTips":["Keep delta.deletedFileRetentionDuration comfortably longer than the pipeline's maximum runtime.","Never run VACUUM concurrently with CDC reads.","Verify worker IAM/credentials can read the table location before launching."],"tags":["java","parquet","file-read","delta-lake"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}