apache/iceberg · error · UncheckedIOException

Failed to plan files for main index

Error message

Failed to plan files for main index

What it means

Thrown by EqualityConvertPlanner.emitMainDataReadCommands when an IOException occurs while enumerating the data files of the main-branch snapshot needed to build the main PK index. It wraps the IOException in an UncheckedIOException because file planning is essential to emit ReadCommand data-file records downstream.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:698

   * them for the configured equality-field set. Existing DVs attached to a data file are loaded by
   * the reader and their positions are skipped. V2 positional deletes are not expected on main; the
   * reader throws if it encounters one. Equality deletes attached to the scan task are skipped
   * during indexing (they are processed via the planner's eq-delete read commands).
   */
  private void emitMainDataReadCommands(Snapshot mainSnapshot) {
    long commitSnapshotId = mainSnapshot.snapshotId();

    try (CloseableIterable<FileScanTask> tasks =
        table.newScan().useSnapshot(commitSnapshotId).planFiles()) {
      for (FileScanTask task : tasks) {
        output.collect(
            new StreamRecord<>(
                ReadCommand.dataFile(
                    task, indexSnapshotId, indexGeneration, dataSequenceNumber(task.file())),
                nextPhaseTs));
      }
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to plan files for main index", e);
    }

    LOG.info(
        "Emitted main data read commands for field IDs {} from snapshot {}.",
        eqFieldIds,
        commitSnapshotId);

    advancePhase();
  }

  /**
   * Emits a phase-end watermark and bumps the phase timestamp. Every phase-emitting method must
   * call this exactly once after its records; the worker uses these watermarks to gate keyed-state
   * transitions. Missing or extra calls silently break ordering.
   */
  private void advancePhase() {
    output.emitWatermark(new Watermark(nextPhaseTs));
    nextPhaseTs++;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the wrapped cause and verify the manifest/data files of the commit snapshot are readable and present.
2. Stop concurrent expireSnapshots/removeOrphanFiles jobs while the conversion job runs.
3. Check FileIO storage credentials, endpoints, and permissions; retry the job after transient network issues.
4. Rerun planning after the table is stable; use snapshot rollback if metadata references missing files.

Example fix

// before
expireSnapshots(table).execute(); // concurrent with conversion job
runEqualityConvert(table);
// after
runEqualityConvert(table);
expireSnapshots(table).execute(); // schedule maintenance sequentially

Snapshot snap = table.snapshotForBranch(targetBranch);
snap.dataManifests(table.io()).forEach(m -> Preconditions.checkArgument(table.io().newInputFile(m.path()).exists(), "missing manifest " + m.path()));

try {
  runEqualityConvertJob(table, cfg);
} catch (UncheckedIOException e) {
  LOG.error("Main-index file planning failed; cause={}", e.getCause(), e);
  // reschedule after transient storage errors are resolved
}

When it happens

Trigger: IOException from opening/iterating the snapshot's manifest contents (transient object-store failure, network error, deleted manifest, permission problem) while emitting data-file ReadCommands from rebuildIndex.

Common situations: Concurrent snapshot expiration removing manifests still referenced by the planner; S3 throttling or HDFS instability; credentials/permission misconfiguration on the FileIO layer.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to read manifest:
• Failed to close manifest reader
• Failed to close entries while caching changes
• Failed to close current writer
• Failed to close dvFileWriter

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/683cd66a9cfd004a. Report an issue: GitHub.