{"record":{"id":"d03d94bb4ea33fd4","repo":"apache/iceberg","slug":"interrupted-while-polling-queue","errorCode":null,"errorMessage":"Interrupted while polling queue","messagePattern":"Interrupted while polling queue","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java","lineNumber":183,"sourceCode":"        key -> {\n          LOG.info(\n              \"running planFiles for {}, startOffset: {}, endOffset: {}\",\n              table().name(),\n              startOffset,\n              endOffset);\n          List<FileScanTask> result = new LinkedList<>();\n          Pair<StreamingOffset, FileScanTask> elem;\n          StreamingOffset currentOffset;\n          boolean shouldTerminate = false;\n          long filesInPlan = 0;\n          long rowsInPlan = 0;\n\n          do {\n            try {\n              elem = queue.pollFirst(QUEUE_POLL_TIMEOUT_MS, TimeUnit.MILLISECONDS);\n            } catch (InterruptedException e) {\n              Thread.currentThread().interrupt();\n              throw new RuntimeException(\"Interrupted while polling queue\", e);\n            }\n\n            if (elem != null) {\n              currentOffset = elem.first();\n              LOG.debug(\"planFiles consumed: {}\", currentOffset);\n              FileScanTask currentTask = elem.second();\n              filesInPlan += 1;\n              long elemRows = currentTask.file().recordCount();\n              rowsInPlan += elemRows;\n              queuedFileCount.decrementAndGet();\n              queuedRowCount.addAndGet(-elemRows);\n              result.add(currentTask);\n\n              // try to peek at the next entry of the queue and see if we should stop\n              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();\n              boolean endOffsetPeek = false;\n              if (nextElem != null) {\n                endOffsetPeek = endOffset.equals(nextElem.first());","sourceCodeStart":165,"sourceCodeEnd":201,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java#L165-L201","documentation":"In AsyncSparkMicroBatchPlanner.planFiles, the thread polls a queue that a background thread fills with planned FileScanTasks. If that poll is interrupted, the code restores the interrupt flag and wraps the InterruptedException in a RuntimeException. This signals the streaming micro-batch planning loop was interrupted externally, typically during query cancellation or shutdown.","triggerScenarios":"Spark interrupting the streaming query while planFiles waits longer than QUEUE_POLL_TIMEOUT_MS for planned tasks; query.stop() or job cancellation racing with batch planning.","commonSituations":"Stopping a Spark structured streaming query, driver shutdown, or user-triggered cancellation while the async planner is still waiting for planned files.","solutions":["Treat as expected during query stop/cancellation; verify the query was intentionally interrupted.","If unexpected, check driver logs for why the streaming query or its executor threads were interrupted.","Inspect background thread failures that may have stalled queue filling, forcing the long poll.","Retry the batch/streaming query once the environment is stable."],"exampleFix":"try {\n  df.writeStream().start();\n} catch (StreamingQueryException e) {\n  if (e.getCause() != null && e.getCause().getMessage().contains(\"Interrupted while polling queue\")) {\n    // query was stopped/cancelled; no action needed\n  }\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  query.awaitTermination();\n} catch (StreamingQueryException e) {\n  if (e.getCause() instanceof RuntimeException\n      && e.getCause().getMessage().contains(\"Interrupted while polling queue\")) {\n    // expected during stop/cancel; log and proceed\n  } else {\n    throw e;\n  }\n}","preventionTips":["Stop streaming queries via query.stop() rather than killing the driver.","Correlate the error with shutdown events before treating it as a bug.","Monitor background planner thread health to avoid stalled queues that lead to long polls."],"tags":["spark","streaming","interruption","threading"],"backgroundTag":"request-timeout","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}