{"record":{"id":"6f06dc59bf479a20","repo":"apache/iceberg","slug":"failed-to-plan-files-for-main-index-6f06dc","errorCode":null,"errorMessage":"Failed to plan files for main index","messagePattern":"Failed to plan files for main index","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java","lineNumber":698,"sourceCode":"   * them for the configured equality-field set. Existing DVs attached to a data file are loaded by\n   * the reader and their positions are skipped. V2 positional deletes are not expected on main; the\n   * reader throws if it encounters one. Equality deletes attached to the scan task are skipped\n   * during indexing (they are processed via the planner's eq-delete read commands).\n   */\n  private void emitMainDataReadCommands(Snapshot mainSnapshot) {\n    long commitSnapshotId = mainSnapshot.snapshotId();\n\n    try (CloseableIterable<FileScanTask> tasks =\n        table.newScan().useSnapshot(commitSnapshotId).planFiles()) {\n      for (FileScanTask task : tasks) {\n        output.collect(\n            new StreamRecord<>(\n                ReadCommand.dataFile(\n                    task, indexSnapshotId, indexGeneration, dataSequenceNumber(task.file())),\n                nextPhaseTs));\n      }\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to plan files for main index\", e);\n    }\n\n    LOG.info(\n        \"Emitted main data read commands for field IDs {} from snapshot {}.\",\n        eqFieldIds,\n        commitSnapshotId);\n\n    advancePhase();\n  }\n\n  /**\n   * Emits a phase-end watermark and bumps the phase timestamp. Every phase-emitting method must\n   * call this exactly once after its records; the worker uses these watermarks to gate keyed-state\n   * transitions. Missing or extra calls silently break ordering.\n   */\n  private void advancePhase() {\n    output.emitWatermark(new Watermark(nextPhaseTs));\n    nextPhaseTs++;","sourceCodeStart":680,"sourceCodeEnd":716,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java#L680-L716","documentation":"emitMainDataReadCommands uses an Iceberg scan over the main/index snapshot to enumerate data files and emit ReadCommand records. An IOException during scan planning (manifest/table metadata reads) is wrapped in UncheckedIOException with the message 'Failed to plan files for main index'. The index-build cycle cannot proceed without this file list.","triggerScenarios":"rebuildIndex -> emitMainDataReadCommands calls table scan planning; underlying manifest or metadata file reads fail due to storage errors, deleted manifests (concurrent expiry), or credential problems.","commonSituations":"Concurrent snapshot expiry removing files mid-plan; S3/GCS throttling or auth failures; HDFS outage; table metadata churn from concurrent commits invalidating file handles.","solutions":["Retry the maintenance cycle — planning failures are often transient storage issues.","Pause expireSnapshots/orphan cleanup while the converter runs.","Verify FileIO credentials and endpoint configuration for the table's storage.","Check the wrapped IOException for the concrete storage error and address it (permissions, throttling, network)."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// pre-flight: confirm the table's current metadata and manifests are readable\ntable.refresh();\ntable.io().newInputFile(table.operations().current().metadataFileLocation()).exists();","typeGuard":null,"tryCatchPattern":"try {\n  rebuildIndex(...);\n} catch (UncheckedIOException e) {\n  if (e.getMessage().equals(\"Failed to plan files for main index\")) {\n    retryWithBackoff(() -> rebuildIndex(...));\n  } else {\n    throw e;\n  }\n}","preventionTips":["Pause snapshot expiry during conversion cycles.","Verify FileIO credentials before long-running jobs.","Add object-store retry config (throttling/5xx) at the FileIO level.","Retry cycles automatically on UncheckedIOException."],"tags":["io","scan-planning","manifest","flink"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}