apache/iceberg · error · UncheckedIOException

Failed to plan files for main index

Error message

Failed to plan files for main index

What it means

This UncheckedIOException wraps an IOException thrown while executing table.newScan().planFiles() in the Iceberg Flink maintenance EqualityConvertPlanner. During the index rebuild phase the planner plans the main branch snapshot's files so each can be turned into a ReadCommand for the equality-delete-to-DV conversion workers. File planning reads manifests from the table's FileIO, so metadata fetch failures (network, missing/corrupt manifests, authentication) surface here.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:698

   * them for the configured equality-field set. Existing DVs attached to a data file are loaded by
   * the reader and their positions are skipped. V2 positional deletes are not expected on main; the
   * reader throws if it encounters one. Equality deletes attached to the scan task are skipped
   * during indexing (they are processed via the planner's eq-delete read commands).
   */
  private void emitMainDataReadCommands(Snapshot mainSnapshot) {
    long commitSnapshotId = mainSnapshot.snapshotId();

    try (CloseableIterable<FileScanTask> tasks =
        table.newScan().useSnapshot(commitSnapshotId).planFiles()) {
      for (FileScanTask task : tasks) {
        output.collect(
            new StreamRecord<>(
                ReadCommand.dataFile(
                    task, indexSnapshotId, indexGeneration, dataSequenceNumber(task.file())),
                nextPhaseTs));
      }
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to plan files for main index", e);
    }

    LOG.info(
        "Emitted main data read commands for field IDs {} from snapshot {}.",
        eqFieldIds,
        commitSnapshotId);

    advancePhase();
  }

  /**
   * Emits a phase-end watermark and bumps the phase timestamp. Every phase-emitting method must
   * call this exactly once after its records; the worker uses these watermarks to gate keyed-state
   * transitions. Missing or extra calls silently break ordering.
   */
  private void advancePhase() {
    output.emitWatermark(new Watermark(nextPhaseTs));
    nextPhaseTs++;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the FileIO/warehouse storage is reachable and credentials are valid from the Flink TaskManager (test reads on the metadata location).
2. Check that snapshot expiration jobs are not concurrently deleting manifests for the snapshot being planned; serialize maintenance triggers via the lock factory.
3. Retry the maintenance cycle — planning is read-only and safe to re-run; inspect the wrapped IOException cause for the real storage error.
4. Validate manifest integrity (e.g., inspect via InspectSnapshots/spark procedures) if corruption is suspected and remove the bad snapshot from the trigger scope.

Example fix

// before
CloseableIterable<FileScanTask> tasks = table.newScan().useSnapshot(commitSnapshotId).planFiles();
// throws UncheckedIOException when storage is down

// after
// wrap the whole rebuildIndex call with retry at the trigger level
try {
  rebuildIndex();
} catch (UncheckedIOException e) {
  LOG.warn("File planning failed, will retry next trigger", e.getCause());
  // re-trigger maintenance instead of failing the job
}

// before triggering maintenance
try (CloseableIterable<FileScanTask> probe = table.newScan().planFiles()) {
  probe.iterator().hasNext(); // forces metadata read
}

try {
  rebuildIndex();
} catch (UncheckedIOException e) {
  LOG.error("Main file planning failed: {}", e.getCause().getMessage());
  // re-trigger on the next maintenance cycle instead of crashing the job
}

When it happens

Trigger: rebuildIndex -> emitMainDataReadCommands calls table.newScan().useSnapshot(commitSnapshotId).planFiles() and the underlying FileIO throws an IOException while reading manifest lists/manifests for the main snapshot.

Common situations: Object store (S3/HDFS) connectivity or permission problems during a maintenance trigger; the snapshot referenced by commitSnapshotId was expired concurrently by another maintenance job so its manifests are gone; corrupt or truncated manifest files; transient IO errors in Hadoop FileIO.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to list partitions of table %s
• Failed to plan files for main index
• Failed to process tasks iterable
• Failed to plan files for main index
• Failed to list partitions of table %s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f5285110cc3be68e. Report an issue: GitHub.