{"record":{"id":"9589442a5e3cbb31","repo":"apache/iceberg","slug":"failed-to-close-task-iterable-958944","errorCode":null,"errorMessage":"Failed to close task iterable","messagePattern":"Failed to close task iterable","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java","lineNumber":207,"sourceCode":"                // read in the current snapshot.\n                if (curFilesAdded == 1 && curRecordCount > maxRows) {\n                  LOG.warn(\n                      \"File {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. \"\n                          + \"This file will be processed entirely to guarantee forward progress. \"\n                          + \"Consider increasing the limit or writing smaller files to avoid unexpected memory usage.\",\n                      task.file().location(),\n                      task.file().recordCount(),\n                      maxRows);\n                }\n                ++curPos;\n                shouldContinueReading = false;\n                break;\n              }\n            }\n            ++curPos;\n          }\n        } catch (IOException ioe) {\n          LOG.warn(\"Failed to close task iterable\", ioe);\n        }\n      }\n      // if the currentSnapShot was also the latestSnapshot then break\n      if (curSnapshot.snapshotId() == latestSnapshotId) {\n        break;\n      }\n\n      // if everything was OK and we consumed complete snapshot then move to next snapshot\n      if (shouldContinueReading) {\n        Snapshot nextValid = nextValidSnapshot(curSnapshot);\n        if (nextValid == null) {\n          // nextValid implies all the remaining snapshots should be skipped.\n          break;\n        }\n        // we found the next available snapshot, continue from there.\n        curSnapshot = nextValid;\n        startPosOfSnapOffset = -1;\n        // if anyhow we are moving to next snapshot we should only scan addedFiles","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java#L189-L225","documentation":"This is a logged warning (not a thrown exception) emitted when closing a task iterable during Spark Structured Streaming micro-batch planning fails with an IOException. The planner iterates snapshots to build the batch, and if closing the underlying iterable's resources (e.g., file handles via FileIO) fails, the failure is swallowed and only warned because planning can still proceed. It signals a resource cleanup problem in the table's IO layer rather than a fatal planning error.","triggerScenarios":"Calling latestOffset on a SparkStreamingScan during Structured Streaming micro-batch planning when CloseableIterable.close() on the task iterable throws IOException — typically underlying HDFS/S3 read failures, network interruptions, or credentials expiring mid-iteration.","commonSituations":"S3/HDFS transient I/O errors during streaming; IAM/STS credentials expiring between batches; filesystem connectivity flaps in long-running streaming queries; cases where the snapshot changed while planning and the iterable close raced with cleanup.","solutions":["Inspect the chained IOException cause in the log to find the real storage-side failure (permissions, network, credentials).","Check cluster/network stability between the driver/executor and the object store or HDFS NameNode.","Refresh or extend credential lifetimes (e.g., longer STS session, instance profile) for long-running streaming jobs.","Upgrade Iceberg/Spark connector versions; transient close failures have been fixed in newer releases.","If it recurs, enable verbose FileIO/S3 client logging to identify the underlying request that fails."],"exampleFix":"// before\n} catch (IOException ioe) {\n  LOG.warn(\"Failed to close task iterable\", ioe);\n}\n// after\n} catch (IOException ioe) {\n  LOG.warn(\"Failed to close task iterable; check storage connectivity/credentials\", ioe);\n  // investigate ioe.getCause() for the underlying storage failure\n}","handlingStrategy":"retry","validationCode":"// verify the table location is reachable before streaming\nspark.read.format(\"iceberg\").load(\"db.tbl\").limit(1).collectAsList();","typeGuard":null,"tryCatchPattern":"// treat as transient: rely on Spark streaming retry; monitor driver logs for the chained IOException cause","preventionTips":["Monitor driver logs for repeated occurrences of this warning","Use stable instance profiles or long-lived credentials for streaming jobs","Keep Spark and Iceberg connector versions aligned","Alert on storage-side error rates (S3 5xx, HDFS RPC timeouts)"],"tags":["spark","streaming","io","fileio"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}