apache/iceberg · warning

Failed to close task iterator

Error message

Failed to close task iterator

What it means

While grouping FileScanTasks by partition in BaseRewriteDataFilesAction.groupTasksByPartition(), an IOException raised while iterating (or implicitly closing) the task iterator is caught and logged as this warning. The returned map may then be incomplete, so some file groups could be silently omitted from the rewrite. Callers should treat repeated occurrences as a sign of degraded rewrite coverage.

Solutions

1. Inspect the logged IOException to find the failing stream/storage location.
2. Fix storage access issues (permissions, connectivity) and re-run the rewrite action.
3. Re-plan the scan (table.refresh() then execute) to get a complete task list.
4. Compare the grouped partition count against expected partitions to detect silently missing groups.

int planned = (int) streamOf(table.newScan().planTasks()).count();
if (groupedTasks.size() < expectedPartitionCount) {
  throw new IllegalStateException("Incomplete task grouping; re-plan the scan before rewriting");
}

try {
  tasksByPartition = groupTasksByPartition(taskIterable.iterator());
} catch (RuntimeException e) {
  LOG.warn("grouping possibly incomplete due to IO error; aborting rewrite", e);
}

When it happens

Trigger: Iterator exhaustion/close of the FileScanTask CloseableIterator throws IOException during groupTasksByPartition() — e.g. an open manifest/data stream fails mid-iteration.

Common situations: Object-store read errors mid-iteration; truncated manifest lists; network drops to storage during planning; Hadoop FS misconfiguration.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to close task iterable
• Ignoring ' ' as such values are constant within a partition
• Ignoring ' ' as such values are constant within a partition
• Already closed files for partition:
• Altering partition keys is not supported yet.

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/c08107eacd7d0749. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java:298

            .collect(Collectors.toList());
    replaceDataFiles(currentDataFiles, addedDataFiles, startingSnapshotId);

    return new RewriteDataFilesActionResult(currentDataFiles, addedDataFiles);
  }

  private Map<StructLikeWrapper, Collection<FileScanTask>> groupTasksByPartition(
      CloseableIterator<FileScanTask> tasksIter) {
    ListMultimap<StructLikeWrapper, FileScanTask> tasksGroupedByPartition =
        Multimaps.newListMultimap(Maps.newHashMap(), Lists::newArrayList);
    StructLikeWrapper partitionWrapper = StructLikeWrapper.forType(spec.partitionType());
    try (CloseableIterator<FileScanTask> iterator = tasksIter) {
      iterator.forEachRemaining(
          task -> {
            StructLikeWrapper structLike = partitionWrapper.copyFor(task.file().partition());
            tasksGroupedByPartition.put(structLike, task);
          });
    } catch (IOException e) {
      LOG.warn("Failed to close task iterator", e);
    }
    return tasksGroupedByPartition.asMap();
  }

  private void replaceDataFiles(
      Iterable<DataFile> deletedDataFiles,
      Iterable<DataFile> addedDataFiles,
      long startingSnapshotId) {
    try {
      doReplace(deletedDataFiles, addedDataFiles, startingSnapshotId);
    } catch (CommitStateUnknownException e) {
      LOG.warn("Commit state unknown, cannot clean up files that may have been committed", e);
      throw e;
    } catch (Exception e) {
      if (e instanceof CleanableFailure) {
        LOG.warn("Failed to commit rewrite, cleaning up rewritten files", e);
        Tasks.foreach(Iterables.transform(addedDataFiles, ContentFile::location))
            .noRetry()

View on GitHub (pinned to 86d9c8fc54)