{"record":{"id":"b0f2f5b92226699d","repo":"apache/beam","slug":"failed-to-read-parquet-footer-for-createreadtasksdofn","errorCode":null,"errorMessage":"Failed to read Parquet footer for ","messagePattern":"Failed to read Parquet footer for ","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/CreateReadTasksDoFn.java","lineNumber":157,"sourceCode":"      out.output(readTask);\n    }\n  }\n\n  private List<Long> getRowGroupSizes(SerializableRow scanFileRow, Configuration conf) {\n    List<Long> sizes = new ArrayList<>();\n    String pathStr = InternalScanFileUtils.getAddFileStatus(scanFileRow).getPath();\n    try {\n      org.apache.hadoop.fs.Path hadoopPath = new org.apache.hadoop.fs.Path(pathStr);\n      org.apache.parquet.hadoop.metadata.ParquetMetadata metadata =\n          org.apache.parquet.hadoop.ParquetFileReader.readFooter(\n              conf,\n              hadoopPath,\n              org.apache.parquet.format.converter.ParquetMetadataConverter.NO_FILTER);\n      for (org.apache.parquet.hadoop.metadata.BlockMetaData block : metadata.getBlocks()) {\n        sizes.add(block.getTotalByteSize());\n      }\n    } catch (java.io.IOException e) {\n      throw new RuntimeException(\"Failed to read Parquet footer for \" + pathStr, e);\n    }\n    return sizes;\n  }\n}\n","sourceCodeStart":139,"sourceCodeEnd":162,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/CreateReadTasksDoFn.java#L139-L162","documentation":"Same family as the CDC variant: CreateReadTasksDoFn.getRowGroupSizes reads the Parquet footer of a Delta data file to compute row-group byte sizes for task splitting, and wraps any IOException reading that footer in a RuntimeException naming the file.","triggerScenarios":"Snapshot data file deleted by VACUUM or rewritten by a concurrent writer before read; missing/wrong storage credentials in the Hadoop Configuration; truncated or corrupt Parquet file; transient S3/GCS/HDFS I/O error.","commonSituations":"Reading an old version/timestamp of a Delta table whose files have since been vacuumed; network blips to object storage; IAM roles lacking GetObject on the table location.","solutions":["Retry the read; transient object-store errors resolve on re-run.","Ensure the requested version's files still exist — avoid reading versions older than delta.deletedFileRetentionDuration.","Fix storage configuration on the worker (credentials, region, endpoint) so the path is readable.","Inspect the printed path with a standalone reader (e.g. parquet-tools) to confirm corruption vs. access problems."],"exampleFix":"// before\nDeltaIO.read().from(\"s3://bucket/t\").withVersion(1); // version 1 files vacuumed\n// after\nDeltaIO.read().from(\"s3://bucket/t\").withVersion(latestAvailableNonVacuumedVersion);","handlingStrategy":"retry","validationCode":"// pre-check readability\norg.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.Path.getFileSystem(hadoopPath, conf);\nif (!fs.exists(hadoopPath)) throw new java.io.FileNotFoundException(hadoopPath.toString());","typeGuard":null,"tryCatchPattern":"try { /* read */ } catch (RuntimeException e) { if (e.getCause() instanceof java.io.IOException && isTransient(e.getCause())) { retryWithBackoff(); } else { throw e; } }","preventionTips":["Don't read versions older than the VACUUM retention window.","Pin Delta Kernel/Hadoop storage configs (credentials, region) shared by writer and readers.","Monitor object-store error rates; use retry-capable filesystem implementations (e.g. S3A retry settings)."],"tags":["java","parquet","file-read","delta-lake"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}