What causes "Interrupted while polling queue" in apache/iceberg?

Usually benign during query termination — verify the streaming query was intentionally stopped or cancelled.

apache/iceberg · warning · RuntimeException

Interrupted while polling queue

Error message

Interrupted while polling queue

What it means

AsyncSparkMicroBatchPlanner.planFiles consumes planned (offset, FileScanTask) pairs from a queue filled by a background thread, polling with a timeout in a loop. If the polling thread is interrupted (Thread.interrupt from task cancellation or query stop), it restores the interrupt flag and wraps the InterruptedException in a RuntimeException with this message.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:183

        key -> {
          LOG.info(
              "running planFiles for {}, startOffset: {}, endOffset: {}",
              table().name(),
              startOffset,
              endOffset);
          List<FileScanTask> result = new LinkedList<>();
          Pair<StreamingOffset, FileScanTask> elem;
          StreamingOffset currentOffset;
          boolean shouldTerminate = false;
          long filesInPlan = 0;
          long rowsInPlan = 0;

          do {
            try {
              elem = queue.pollFirst(QUEUE_POLL_TIMEOUT_MS, TimeUnit.MILLISECONDS);
            } catch (InterruptedException e) {
              Thread.currentThread().interrupt();
              throw new RuntimeException("Interrupted while polling queue", e);
            }

            if (elem != null) {
              currentOffset = elem.first();
              LOG.debug("planFiles consumed: {}", currentOffset);
              FileScanTask currentTask = elem.second();
              filesInPlan += 1;
              long elemRows = currentTask.file().recordCount();
              rowsInPlan += elemRows;
              queuedFileCount.decrementAndGet();
              queuedRowCount.addAndGet(-elemRows);
              result.add(currentTask);

              // try to peek at the next entry of the queue and see if we should stop
              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());

View on GitHub (pinned to 86d9c8fc54)

Solutions

Usually benign during query termination — verify the streaming query was intentionally stopped or cancelled.
If unexpected, check for code calling Thread.interrupt() on Spark task threads (custom listeners, watchdogs).
Retry/resume the streaming query from the last committed checkpoint; offsets are durable.
If it recurs without cancellation, report/inspect the background planner thread for deadlock keeping the queue empty.

Defensive patterns

Strategy: try-catch

Try / catch

try {
    query.processAllAvailable();
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().contains("Interrupted while polling queue")) {
        // expected during stop(); re-interrupt and exit cleanly
        Thread.currentThread().interrupt();
    }
}

Prevention

Always stop streaming queries via streamingQuery.stop() and treat interrupt-based exceptions as normal shutdown
Avoid custom code that interrupts Spark execution threads
Rely on checkpoints so interrupted queries resume cleanly

When it happens

Trigger: A Spark structured streaming query reading an Iceberg table via the async micro-batch planner is cancelled or stopped while planFiles is blocked waiting on the queue; Spark's interrupt handling calls Thread.interrupt during query termination or executor shutdown.

Common situations: streamingQuery.stop() racing with an in-flight micro-batch; job cancellation from a scheduler timeout; executor shutdown/preemption in YARN/K8s while planning; Ctrl-C on a streaming query in a notebook.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/da09d76c821067e2. Report an issue: GitHub.