What causes "Interrupted while waiting for tasks to finish" in apache/iceberg?

Avoid interrupting threads running Tasks.waitFor() until commits complete; cancel at a safe boundary

Interrupted while waiting for tasks to finish

Error message

Interrupted while waiting for tasks to finish

What it means

Tasks.waitFor() polls task progress with 10ms sleeps; the sleeping thread was interrupted before all futures finished. All in-flight futures are cancelled with cancel(true), the interrupt flag is restored, and a RuntimeException wrapping InterruptedException is thrown.

Solutions

Avoid interrupting threads running Tasks.waitFor() until commits complete; cancel at a safe boundary
Check that the owning executor is not shut down prematurely (shutdownNow vs shutdown after task completion)
Handle the RuntimeException as cancellation: expect partial work and re-run idempotently
Increase task-level retry/failure handling so cancellations are less frequent

Example fix

// before
executor.shutdownNow(); // interrupts in-flight Tasks.waitFor
// after
executor.shutdown();
executor.awaitTermination(5, TimeUnit.MINUTES);

Defensive patterns

Strategy: try-catch

Try / catch

try {
  Tasks.foreach(items).executeWith(pool).stopOnFailure().run();
} catch (RuntimeException e) {
  if (e.getCause() instanceof InterruptedException) {
    // futures were cancelled(true); re-run idempotently if needed
    handleCancellation();
  }
}

Prevention

Avoid interrupting threads performing Iceberg commits mid-flight
Use executor.shutdown() + awaitTermination instead of shutdownNow() during work
Design tasks to be idempotent so cancellation recovery is safe

When it happens

Trigger: The thread waiting on task completion is interrupted mid-run — executor shutdownNow(), Spark/Flink task cancellation, or manual Thread.interrupt() while Tasks-based parallel work (e.g. commits, deletes) is still executing.

Common situations: Cancelling a long-running Spark stage that performs Iceberg commits; executor lifecycle mismanagement in custom actions; operator-triggered job restarts.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/394e82b58e2befc8. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/util/Tasks.java:524

              }
              throw (Error) cause;
            }

            if (cause != null) {
              uncaught.add(e);
            }

            LOG.warn("Task threw uncaught exception", cause);
          }
        }

        return uncaught;

      } else {
        try {
          Thread.sleep(10);
        } catch (InterruptedException e) {
          LOG.warn("Interrupted while waiting for tasks to finish", e);

          for (Future<?> future : futures) {
            future.cancel(true);
          }
          Thread.currentThread().interrupt();
          throw new RuntimeException(e);
        }
      }
    }
  }

  /** A range, [ 0, size ) */
  private static class Range implements Iterable<Integer> {
    private int size;

    Range(int size) {
      this.size = size;
    }

View on GitHub (pinned to 86d9c8fc54)