apache/iceberg · warning

Failed to close task iterable

Error message

Failed to close task iterable

What it means

During micro-batch planning the planner iterates task iterables (CloseableIterable) per snapshot; if closing such an iterable throws an IOException it is caught and only logged as a warning rather than failing the query. Planning continues, but the underlying resource (e.g. file handle) may not have been released cleanly.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java:207

                // read in the current snapshot.
                if (curFilesAdded == 1 && curRecordCount > maxRows) {
                  LOG.warn(
                      "File {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. "
                          + "This file will be processed entirely to guarantee forward progress. "
                          + "Consider increasing the limit or writing smaller files to avoid unexpected memory usage.",
                      task.file().location(),
                      task.file().recordCount(),
                      maxRows);
                }
                ++curPos;
                shouldContinueReading = false;
                break;
              }
            }
            ++curPos;
          }
        } catch (IOException ioe) {
          LOG.warn("Failed to close task iterable", ioe);
        }
      }
      // if the currentSnapShot was also the latestSnapshot then break
      if (curSnapshot.snapshotId() == latestSnapshotId) {
        break;
      }

      // if everything was OK and we consumed complete snapshot then move to next snapshot
      if (shouldContinueReading) {
        Snapshot nextValid = nextValidSnapshot(curSnapshot);
        if (nextValid == null) {
          // nextValid implies all the remaining snapshots should be skipped.
          break;
        }
        // we found the next available snapshot, continue from there.
        curSnapshot = nextValid;
        startPosOfSnapOffset = -1;
        // if anyhow we are moving to next snapshot we should only scan addedFiles

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the chained IOException in the log for the underlying storage error and address it (connectivity, permissions, throttling)
2. Increase file descriptor limits / connection pool size if handles are exhausted
3. Retry the streaming query; the error is logged, not propagated, so verify planning results are correct

try {
  // streaming read / batch planning
} catch (Exception e) {
  if (e.getCause() instanceof java.io.IOException) {
    // inspect cause, retry planning or restart stream
  }
}

When it happens

Trigger: An IOException thrown from CloseableIterable.close() while iterating manifest/data-file task lists inside latestOffset — usually an underlying filesystem or network error surfaced during close.

Common situations: S3/HDFS transient errors or timeouts during streaming planning; heavy concurrent streaming queries exhausting file descriptors.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Table refresh failed
• Failed writing offset to: ${initialOffsetLocation}
• Failed reading offset from: ${initialOffsetLocation}
• Queue filling failed
• Failed to close changelog scan: + scan

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/08708dbe28e94a1b. Report an issue: GitHub.