apache/iceberg · error · RuntimeException

Interrupted while polling queue

Error message

Interrupted while polling queue

What it means

In AsyncSparkMicroBatchPlanner.planFiles, the thread polls a queue that a background thread fills with planned FileScanTasks. If that poll is interrupted, the code restores the interrupt flag and wraps the InterruptedException in a RuntimeException. This signals the streaming micro-batch planning loop was interrupted externally, typically during query cancellation or shutdown.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:183

        key -> {
          LOG.info(
              "running planFiles for {}, startOffset: {}, endOffset: {}",
              table().name(),
              startOffset,
              endOffset);
          List<FileScanTask> result = new LinkedList<>();
          Pair<StreamingOffset, FileScanTask> elem;
          StreamingOffset currentOffset;
          boolean shouldTerminate = false;
          long filesInPlan = 0;
          long rowsInPlan = 0;

          do {
            try {
              elem = queue.pollFirst(QUEUE_POLL_TIMEOUT_MS, TimeUnit.MILLISECONDS);
            } catch (InterruptedException e) {
              Thread.currentThread().interrupt();
              throw new RuntimeException("Interrupted while polling queue", e);
            }

            if (elem != null) {
              currentOffset = elem.first();
              LOG.debug("planFiles consumed: {}", currentOffset);
              FileScanTask currentTask = elem.second();
              filesInPlan += 1;
              long elemRows = currentTask.file().recordCount();
              rowsInPlan += elemRows;
              queuedFileCount.decrementAndGet();
              queuedRowCount.addAndGet(-elemRows);
              result.add(currentTask);

              // try to peek at the next entry of the queue and see if we should stop
              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Treat as expected during query stop/cancellation; verify the query was intentionally interrupted.
2. If unexpected, check driver logs for why the streaming query or its executor threads were interrupted.
3. Inspect background thread failures that may have stalled queue filling, forcing the long poll.
4. Retry the batch/streaming query once the environment is stable.

Example fix

try {
  df.writeStream().start();
} catch (StreamingQueryException e) {
  if (e.getCause() != null && e.getCause().getMessage().contains("Interrupted while polling queue")) {
    // query was stopped/cancelled; no action needed
  }
}

try {
  query.awaitTermination();
} catch (StreamingQueryException e) {
  if (e.getCause() instanceof RuntimeException
      && e.getCause().getMessage().contains("Interrupted while polling queue")) {
    // expected during stop/cancel; log and proceed
  } else {
    throw e;
  }
}

When it happens

Trigger: Spark interrupting the streaming query while planFiles waits longer than QUEUE_POLL_TIMEOUT_MS for planned tasks; query.stop() or job cancellation racing with batch planning.

Common situations: Stopping a Spark structured streaming query, driver shutdown, or user-triggered cancellation while the async planner is still waiting for planned files.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Queue filling failed
• Interrupted while polling queue
• Interrupted while polling queue
• Interrupted in SQL query
• Interrupted during commit

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d03d94bb4ea33fd4. Report an issue: GitHub.