{"record":{"id":"08708dbe28e94a1b","repo":"apache/iceberg","slug":"failed-to-close-task-iterable-08708d","errorCode":null,"errorMessage":"Failed to close task iterable","messagePattern":"Failed to close task iterable","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java","lineNumber":207,"sourceCode":"                // read in the current snapshot.\n                if (curFilesAdded == 1 && curRecordCount > maxRows) {\n                  LOG.warn(\n                      \"File {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. \"\n                          + \"This file will be processed entirely to guarantee forward progress. \"\n                          + \"Consider increasing the limit or writing smaller files to avoid unexpected memory usage.\",\n                      task.file().location(),\n                      task.file().recordCount(),\n                      maxRows);\n                }\n                ++curPos;\n                shouldContinueReading = false;\n                break;\n              }\n            }\n            ++curPos;\n          }\n        } catch (IOException ioe) {\n          LOG.warn(\"Failed to close task iterable\", ioe);\n        }\n      }\n      // if the currentSnapShot was also the latestSnapshot then break\n      if (curSnapshot.snapshotId() == latestSnapshotId) {\n        break;\n      }\n\n      // if everything was OK and we consumed complete snapshot then move to next snapshot\n      if (shouldContinueReading) {\n        Snapshot nextValid = nextValidSnapshot(curSnapshot);\n        if (nextValid == null) {\n          // nextValid implies all the remaining snapshots should be skipped.\n          break;\n        }\n        // we found the next available snapshot, continue from there.\n        curSnapshot = nextValid;\n        startPosOfSnapOffset = -1;\n        // if anyhow we are moving to next snapshot we should only scan addedFiles","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java#L189-L225","documentation":"During micro-batch planning the planner iterates task iterables (CloseableIterable) per snapshot; if closing such an iterable throws an IOException it is caught and only logged as a warning rather than failing the query. Planning continues, but the underlying resource (e.g. file handle) may not have been released cleanly.","triggerScenarios":"An IOException thrown from CloseableIterable.close() while iterating manifest/data-file task lists inside latestOffset — usually an underlying filesystem or network error surfaced during close.","commonSituations":"S3/HDFS transient errors or timeouts during streaming planning; heavy concurrent streaming queries exhausting file descriptors.","solutions":["Inspect the chained IOException in the log for the underlying storage error and address it (connectivity, permissions, throttling)","Increase file descriptor limits / connection pool size if handles are exhausted","Retry the streaming query; the error is logged, not propagated, so verify planning results are correct"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  // streaming read / batch planning\n} catch (Exception e) {\n  if (e.getCause() instanceof java.io.IOException) {\n    // inspect cause, retry planning or restart stream\n  }\n}","preventionTips":["Ensure stable connectivity to object storage from executors","Raise file-descriptor limits on nodes","Watch for fd leaks under long-running streams"],"tags":["spark","streaming","io","resource-cleanup"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}