apache/iceberg · error · UncheckedIOException

Failed to plan files for main index

Error message

Failed to plan files for main index

What it means

Wrapped IOException thrown while planning data files for the main index rebuild (emitMainDataReadCommands). Reading the table's data-file/manifest structure to emit ReadCommands failed with an IOException, converted to UncheckedIOException for the streaming pipeline. The snapshot metadata was readable enough to start but file/manifest IO failed during planning.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:698

   * them for the configured equality-field set. Existing DVs attached to a data file are loaded by
   * the reader and their positions are skipped. V2 positional deletes are not expected on main; the
   * reader throws if it encounters one. Equality deletes attached to the scan task are skipped
   * during indexing (they are processed via the planner's eq-delete read commands).
   */
  private void emitMainDataReadCommands(Snapshot mainSnapshot) {
    long commitSnapshotId = mainSnapshot.snapshotId();

    try (CloseableIterable<FileScanTask> tasks =
        table.newScan().useSnapshot(commitSnapshotId).planFiles()) {
      for (FileScanTask task : tasks) {
        output.collect(
            new StreamRecord<>(
                ReadCommand.dataFile(
                    task, indexSnapshotId, indexGeneration, dataSequenceNumber(task.file())),
                nextPhaseTs));
      }
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to plan files for main index", e);
    }

    LOG.info(
        "Emitted main data read commands for field IDs {} from snapshot {}.",
        eqFieldIds,
        commitSnapshotId);

    advancePhase();
  }

  /**
   * Emits a phase-end watermark and bumps the phase timestamp. Every phase-emitting method must
   * call this exactly once after its records; the worker uses these watermarks to gate keyed-state
   * transitions. Missing or extra calls silently break ordering.
   */
  private void advancePhase() {
    output.emitWatermark(new Watermark(nextPhaseTs));
    nextPhaseTs++;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect e.getCause() for the underlying IOException and resolve storage connectivity/credential issues.
2. Ensure snapshot expiration/cleanup does not run concurrently with the maintenance cycle.
3. Re-run the maintenance job - planning is restartable from the new cycle.
4. Configure FileIO/client retries and timeouts for transient storage errors.

Example fix

// before: expiring snapshots in the same pipeline while conversion runs
expireSnapshots(table).execute(); rewriteDvs(table).execute();
// after
rewriteDvs(table).execute(); expireSnapshots(table).execute();

// pre-flight: confirm the commit snapshot's manifest lists open successfully
Snapshot s = table.snapshot(commitSnapshotId);
s.allManifests(table.io()).forEach(m -> checkState(table.io().newInputFile(m.path()).exists(), "missing " + m.path()));

boolean snapshotPlanReadable(Table table, long snapshotId) {
  try {
    table.snapshot(snapshotId).allManifests(table.io()).forEach(m -> table.io().newInputFile(m.path()).exists());
    return true;
  } catch (Exception e) { return false; }
}

try {
  rebuildIndex(...);
} catch (UncheckedIOException e) {
  if (isTransient(e.getCause())) backoffAndRetry(3);
  else throw e;
}

When it happens

Trigger: Manifest or manifest-list IO failure while TableScan/iteration inside rebuildIndex opens snapshot content of the commit snapshot; transient object-storage errors, missing manifests, or credentials problems.

Common situations: S3/GCS throttling or expired credentials mid-plan; concurrent snapshot expiration deleted files being planned; network partition between the Flink taskmanager and object storage.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to list partitions of table %s
• Failed to plan files for main index
• Failed to process tasks iterable
• Failed to plan files for main index
• Failed to list partitions of table %s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/e9e6b88c077152c9. Report an issue: GitHub.