apache/iceberg · error · UncheckedIOException

Failed to plan files for main index

Error message

Failed to plan files for main index

What it means

emitMainDataReadCommands uses an Iceberg scan over the main/index snapshot to enumerate data files and emit ReadCommand records. An IOException during scan planning (manifest/table metadata reads) is wrapped in UncheckedIOException with the message 'Failed to plan files for main index'. The index-build cycle cannot proceed without this file list.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:698

   * them for the configured equality-field set. Existing DVs attached to a data file are loaded by
   * the reader and their positions are skipped. V2 positional deletes are not expected on main; the
   * reader throws if it encounters one. Equality deletes attached to the scan task are skipped
   * during indexing (they are processed via the planner's eq-delete read commands).
   */
  private void emitMainDataReadCommands(Snapshot mainSnapshot) {
    long commitSnapshotId = mainSnapshot.snapshotId();

    try (CloseableIterable<FileScanTask> tasks =
        table.newScan().useSnapshot(commitSnapshotId).planFiles()) {
      for (FileScanTask task : tasks) {
        output.collect(
            new StreamRecord<>(
                ReadCommand.dataFile(
                    task, indexSnapshotId, indexGeneration, dataSequenceNumber(task.file())),
                nextPhaseTs));
      }
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to plan files for main index", e);
    }

    LOG.info(
        "Emitted main data read commands for field IDs {} from snapshot {}.",
        eqFieldIds,
        commitSnapshotId);

    advancePhase();
  }

  /**
   * Emits a phase-end watermark and bumps the phase timestamp. Every phase-emitting method must
   * call this exactly once after its records; the worker uses these watermarks to gate keyed-state
   * transitions. Missing or extra calls silently break ordering.
   */
  private void advancePhase() {
    output.emitWatermark(new Watermark(nextPhaseTs));
    nextPhaseTs++;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Retry the maintenance cycle — planning failures are often transient storage issues.
2. Pause expireSnapshots/orphan cleanup while the converter runs.
3. Verify FileIO credentials and endpoint configuration for the table's storage.
4. Check the wrapped IOException for the concrete storage error and address it (permissions, throttling, network).

// pre-flight: confirm the table's current metadata and manifests are readable
table.refresh();
table.io().newInputFile(table.operations().current().metadataFileLocation()).exists();

try {
  rebuildIndex(...);
} catch (UncheckedIOException e) {
  if (e.getMessage().equals("Failed to plan files for main index")) {
    retryWithBackoff(() -> rebuildIndex(...));
  } else {
    throw e;
  }
}

When it happens

Trigger: rebuildIndex -> emitMainDataReadCommands calls table scan planning; underlying manifest or metadata file reads fail due to storage errors, deleted manifests (concurrent expiry), or credential problems.

Common situations: Concurrent snapshot expiry removing files mid-plan; S3/GCS throttling or auth failures; HDFS outage; table metadata churn from concurrent commits invalidating file handles.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to list partitions of table %s
• Failed to read manifest: <manifest.path()>
• Failed to plan files for main index
• Failed to read manifest:
• Failed to plan files for main index

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6f06dc59bf479a20. Report an issue: GitHub.