{"record":{"id":"683cd66a9cfd004a","repo":"apache/iceberg","slug":"failed-to-plan-files-for-main-index-683cd6","errorCode":null,"errorMessage":"Failed to plan files for main index","messagePattern":"Failed to plan files for main index","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java","lineNumber":698,"sourceCode":"   * them for the configured equality-field set. Existing DVs attached to a data file are loaded by\n   * the reader and their positions are skipped. V2 positional deletes are not expected on main; the\n   * reader throws if it encounters one. Equality deletes attached to the scan task are skipped\n   * during indexing (they are processed via the planner's eq-delete read commands).\n   */\n  private void emitMainDataReadCommands(Snapshot mainSnapshot) {\n    long commitSnapshotId = mainSnapshot.snapshotId();\n\n    try (CloseableIterable<FileScanTask> tasks =\n        table.newScan().useSnapshot(commitSnapshotId).planFiles()) {\n      for (FileScanTask task : tasks) {\n        output.collect(\n            new StreamRecord<>(\n                ReadCommand.dataFile(\n                    task, indexSnapshotId, indexGeneration, dataSequenceNumber(task.file())),\n                nextPhaseTs));\n      }\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to plan files for main index\", e);\n    }\n\n    LOG.info(\n        \"Emitted main data read commands for field IDs {} from snapshot {}.\",\n        eqFieldIds,\n        commitSnapshotId);\n\n    advancePhase();\n  }\n\n  /**\n   * Emits a phase-end watermark and bumps the phase timestamp. Every phase-emitting method must\n   * call this exactly once after its records; the worker uses these watermarks to gate keyed-state\n   * transitions. Missing or extra calls silently break ordering.\n   */\n  private void advancePhase() {\n    output.emitWatermark(new Watermark(nextPhaseTs));\n    nextPhaseTs++;","sourceCodeStart":680,"sourceCodeEnd":716,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java#L680-L716","documentation":"Thrown by EqualityConvertPlanner.emitMainDataReadCommands when an IOException occurs while enumerating the data files of the main-branch snapshot needed to build the main PK index. It wraps the IOException in an UncheckedIOException because file planning is essential to emit ReadCommand data-file records downstream.","triggerScenarios":"IOException from opening/iterating the snapshot's manifest contents (transient object-store failure, network error, deleted manifest, permission problem) while emitting data-file ReadCommands from rebuildIndex.","commonSituations":"Concurrent snapshot expiration removing manifests still referenced by the planner; S3 throttling or HDFS instability; credentials/permission misconfiguration on the FileIO layer.","solutions":["Inspect the wrapped cause and verify the manifest/data files of the commit snapshot are readable and present.","Stop concurrent expireSnapshots/removeOrphanFiles jobs while the conversion job runs.","Check FileIO storage credentials, endpoints, and permissions; retry the job after transient network issues.","Rerun planning after the table is stable; use snapshot rollback if metadata references missing files."],"exampleFix":"// before\nexpireSnapshots(table).execute(); // concurrent with conversion job\nrunEqualityConvert(table);\n// after\nrunEqualityConvert(table);\nexpireSnapshots(table).execute(); // schedule maintenance sequentially","handlingStrategy":"try-catch","validationCode":"Snapshot snap = table.snapshotForBranch(targetBranch);\nsnap.dataManifests(table.io()).forEach(m -> Preconditions.checkArgument(table.io().newInputFile(m.path()).exists(), \"missing manifest \" + m.path()));","typeGuard":null,"tryCatchPattern":"try {\n  runEqualityConvertJob(table, cfg);\n} catch (UncheckedIOException e) {\n  LOG.error(\"Main-index file planning failed; cause={}\", e.getCause(), e);\n  // reschedule after transient storage errors are resolved\n}","preventionTips":["Avoid concurrent snapshot expiration while the job runs","Configure retries for the object store (S3 client retry, HDFS client timeout)","Verify storage permissions for the FileIO implementation","Retrigger the job after transient network failures rather than ignoring it"],"tags":["io","planning","flink","unchecked-exception"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}