apache/iceberg · warning

Failed to close task iterable

Error message

Failed to close task iterable

What it means

This is a logged warning (not a thrown exception) emitted when closing a task iterable during Spark Structured Streaming micro-batch planning fails with an IOException. The planner iterates snapshots to build the batch, and if closing the underlying iterable's resources (e.g., file handles via FileIO) fails, the failure is swallowed and only warned because planning can still proceed. It signals a resource cleanup problem in the table's IO layer rather than a fatal planning error.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java:207

                // read in the current snapshot.
                if (curFilesAdded == 1 && curRecordCount > maxRows) {
                  LOG.warn(
                      "File {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. "
                          + "This file will be processed entirely to guarantee forward progress. "
                          + "Consider increasing the limit or writing smaller files to avoid unexpected memory usage.",
                      task.file().location(),
                      task.file().recordCount(),
                      maxRows);
                }
                ++curPos;
                shouldContinueReading = false;
                break;
              }
            }
            ++curPos;
          }
        } catch (IOException ioe) {
          LOG.warn("Failed to close task iterable", ioe);
        }
      }
      // if the currentSnapShot was also the latestSnapshot then break
      if (curSnapshot.snapshotId() == latestSnapshotId) {
        break;
      }

      // if everything was OK and we consumed complete snapshot then move to next snapshot
      if (shouldContinueReading) {
        Snapshot nextValid = nextValidSnapshot(curSnapshot);
        if (nextValid == null) {
          // nextValid implies all the remaining snapshots should be skipped.
          break;
        }
        // we found the next available snapshot, continue from there.
        curSnapshot = nextValid;
        startPosOfSnapOffset = -1;
        // if anyhow we are moving to next snapshot we should only scan addedFiles

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the chained IOException cause in the log to find the real storage-side failure (permissions, network, credentials).
2. Check cluster/network stability between the driver/executor and the object store or HDFS NameNode.
3. Refresh or extend credential lifetimes (e.g., longer STS session, instance profile) for long-running streaming jobs.
4. Upgrade Iceberg/Spark connector versions; transient close failures have been fixed in newer releases.
5. If it recurs, enable verbose FileIO/S3 client logging to identify the underlying request that fails.

Example fix

// before
} catch (IOException ioe) {
  LOG.warn("Failed to close task iterable", ioe);
}
// after
} catch (IOException ioe) {
  LOG.warn("Failed to close task iterable; check storage connectivity/credentials", ioe);
  // investigate ioe.getCause() for the underlying storage failure
}

// verify the table location is reachable before streaming
spark.read.format("iceberg").load("db.tbl").limit(1).collectAsList();

// treat as transient: rely on Spark streaming retry; monitor driver logs for the chained IOException cause

When it happens

Trigger: Calling latestOffset on a SparkStreamingScan during Structured Streaming micro-batch planning when CloseableIterable.close() on the task iterable throws IOException — typically underlying HDFS/S3 read failures, network interruptions, or credentials expiring mid-iteration.

Common situations: S3/HDFS transient I/O errors during streaming; IAM/STS credentials expiring between batches; filesystem connectivity flaps in long-running streaming queries; cases where the snapshot changed while planning and the iterable close raced with cleanup.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Table refresh failed
• Failed writing offset to: ${initialOffsetLocation}
• Failed reading offset from: ${initialOffsetLocation}
• Queue filling failed
• Failed to read StreamingOffset from json

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/9589442a5e3cbb31. Report an issue: GitHub.