{"record":{"id":"e9e6b88c077152c9","repo":"apache/iceberg","slug":"failed-to-plan-files-for-main-index","errorCode":null,"errorMessage":"Failed to plan files for main index","messagePattern":"Failed to plan files for main index","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java","lineNumber":698,"sourceCode":"   * them for the configured equality-field set. Existing DVs attached to a data file are loaded by\n   * the reader and their positions are skipped. V2 positional deletes are not expected on main; the\n   * reader throws if it encounters one. Equality deletes attached to the scan task are skipped\n   * during indexing (they are processed via the planner's eq-delete read commands).\n   */\n  private void emitMainDataReadCommands(Snapshot mainSnapshot) {\n    long commitSnapshotId = mainSnapshot.snapshotId();\n\n    try (CloseableIterable<FileScanTask> tasks =\n        table.newScan().useSnapshot(commitSnapshotId).planFiles()) {\n      for (FileScanTask task : tasks) {\n        output.collect(\n            new StreamRecord<>(\n                ReadCommand.dataFile(\n                    task, indexSnapshotId, indexGeneration, dataSequenceNumber(task.file())),\n                nextPhaseTs));\n      }\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to plan files for main index\", e);\n    }\n\n    LOG.info(\n        \"Emitted main data read commands for field IDs {} from snapshot {}.\",\n        eqFieldIds,\n        commitSnapshotId);\n\n    advancePhase();\n  }\n\n  /**\n   * Emits a phase-end watermark and bumps the phase timestamp. Every phase-emitting method must\n   * call this exactly once after its records; the worker uses these watermarks to gate keyed-state\n   * transitions. Missing or extra calls silently break ordering.\n   */\n  private void advancePhase() {\n    output.emitWatermark(new Watermark(nextPhaseTs));\n    nextPhaseTs++;","sourceCodeStart":680,"sourceCodeEnd":716,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java#L680-L716","documentation":"Wrapped IOException thrown while planning data files for the main index rebuild (emitMainDataReadCommands). Reading the table's data-file/manifest structure to emit ReadCommands failed with an IOException, converted to UncheckedIOException for the streaming pipeline. The snapshot metadata was readable enough to start but file/manifest IO failed during planning.","triggerScenarios":"Manifest or manifest-list IO failure while TableScan/iteration inside rebuildIndex opens snapshot content of the commit snapshot; transient object-storage errors, missing manifests, or credentials problems.","commonSituations":"S3/GCS throttling or expired credentials mid-plan; concurrent snapshot expiration deleted files being planned; network partition between the Flink taskmanager and object storage.","solutions":["Inspect e.getCause() for the underlying IOException and resolve storage connectivity/credential issues.","Ensure snapshot expiration/cleanup does not run concurrently with the maintenance cycle.","Re-run the maintenance job - planning is restartable from the new cycle.","Configure FileIO/client retries and timeouts for transient storage errors."],"exampleFix":"// before: expiring snapshots in the same pipeline while conversion runs\nexpireSnapshots(table).execute(); rewriteDvs(table).execute();\n// after\nrewriteDvs(table).execute(); expireSnapshots(table).execute();","handlingStrategy":"retry","validationCode":"// pre-flight: confirm the commit snapshot's manifest lists open successfully\nSnapshot s = table.snapshot(commitSnapshotId);\ns.allManifests(table.io()).forEach(m -> checkState(table.io().newInputFile(m.path()).exists(), \"missing \" + m.path()));","typeGuard":"boolean snapshotPlanReadable(Table table, long snapshotId) {\n  try {\n    table.snapshot(snapshotId).allManifests(table.io()).forEach(m -> table.io().newInputFile(m.path()).exists());\n    return true;\n  } catch (Exception e) { return false; }\n}","tryCatchPattern":"try {\n  rebuildIndex(...);\n} catch (UncheckedIOException e) {\n  if (isTransient(e.getCause())) backoffAndRetry(3);\n  else throw e;\n}","preventionTips":["Pause snapshot expiration during index rebuild cycles.","Configure S3/GCS clients with retries and adequate timeouts on taskmanagers.","Use stable, role-based storage credentials instead of short-lived tokens.","Verify warehouse integrity after any manual file cleanup."],"tags":["flink","io","scan-planning"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}