apache/iceberg · warning · RuntimeException

Interrupted while polling queue

Error message

Interrupted while polling queue

What it means

AsyncSparkMicroBatchPlanner.planFiles consumes planned file-scan tasks from a bounded blocking queue, polling with a timeout in a loop. If the polling thread is interrupted while waiting for the next element, it re-interrupts the thread and throws a RuntimeException('Interrupted while polling queue', e).

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:183

        key -> {
          LOG.info(
              "running planFiles for {}, startOffset: {}, endOffset: {}",
              table().name(),
              startOffset,
              endOffset);
          List<FileScanTask> result = new LinkedList<>();
          Pair<StreamingOffset, FileScanTask> elem;
          StreamingOffset currentOffset;
          boolean shouldTerminate = false;
          long filesInPlan = 0;
          long rowsInPlan = 0;

          do {
            try {
              elem = queue.pollFirst(QUEUE_POLL_TIMEOUT_MS, TimeUnit.MILLISECONDS);
            } catch (InterruptedException e) {
              Thread.currentThread().interrupt();
              throw new RuntimeException("Interrupted while polling queue", e);
            }

            if (elem != null) {
              currentOffset = elem.first();
              LOG.debug("planFiles consumed: {}", currentOffset);
              FileScanTask currentTask = elem.second();
              filesInPlan += 1;
              long elemRows = currentTask.file().recordCount();
              rowsInPlan += elemRows;
              queuedFileCount.decrementAndGet();
              queuedRowCount.addAndGet(-elemRows);
              result.add(currentTask);

              // try to peek at the next entry of the queue and see if we should stop
              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. If the interruption was intentional (query stop/shutdown), no fix needed — this is expected cancellation propagation; handle or ignore the resulting failure in streaming termination logic.
2. Retry the micro-batch after restart; streaming checkpoints make replanning safe.
3. Reduce batch planning cost (smaller snapshots, more frequent commits, adequate executor resources) so planning finishes before shutdown windows.

// streaming side: let Spark retry the batch; wrap stop() calls
try {
  query.awaitTermination()
} catch {
  case e: StreamingQueryException if e.getCause.getMessage.contains("Interrupted while polling") =>
    logger.info("Query stopped mid-plan; restart from checkpoint")
}

When it happens

Trigger: A streaming query consuming an Iceberg table (micro-batch planning) whose planning thread is interrupted — typically by query.stop(), cancellation of the job, or Spark executor shutdown while a batch is being planned.

Common situations: Stopping a Spark structured streaming query mid-batch; executor kill / decommissioning; cancelJobGroup during long planning of a huge snapshot; application shutdown.

Related errors

• Interrupted while polling queue
• Interrupted while polling queue
• Interrupted when waiting for deletions to complete
• Interrupted while waiting for array pool entry
• Interrupted in call to listTables

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3df87498b51f2f6d. Report an issue: GitHub.