{"record":{"id":"8aca7bcdfbb11129","repo":"apache/iceberg","slug":"queue-filling-failed","errorCode":null,"errorMessage":"Queue filling failed","messagePattern":"Queue filling failed","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java","lineNumber":220,"sourceCode":"              }\n              // end offset may be synthetic and not exist in the queue\n              boolean endOffsetSynthetic =\n                  currentOffset.snapshotId() == endOffset.snapshotId()\n                      && (currentOffset.position() + 1) == endOffset.position();\n              shouldTerminate = endOffsetPeek || endOffsetSynthetic;\n            } else {\n              LOG.trace(\"planFiles hasn't reached {}, waiting\", endOffset);\n            }\n          } while (!shouldTerminate\n              && refreshFailedThrowable == null\n              && fillQueueFailedThrowable == null);\n\n          if (refreshFailedThrowable != null) {\n            throw new RuntimeException(\"Table refresh failed\", refreshFailedThrowable);\n          }\n\n          if (fillQueueFailedThrowable != null) {\n            throw new RuntimeException(\"Queue filling failed\", fillQueueFailedThrowable);\n          }\n\n          LOG.info(\n              \"completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}\",\n              table().name(),\n              startOffset,\n              endOffset,\n              filesInPlan,\n              rowsInPlan);\n          return result;\n        });\n  }\n\n  /**\n   * This needs to be non destructive on the queue as spark could call this multiple times. Each\n   * time, depending on the table state it could return something different\n   *\n   * @param startOffset the starting offset of the next microbatch","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java#L202-L238","documentation":"After the planning loop terminates, planFiles rethrows fillQueueFailedThrowable as \"Queue filling failed\". The background thread that consumes planned tasks into the queue failed, so the batch cannot be completed even though refresh succeeded.","triggerScenarios":"The queue-filling background task throws while iterating planned FileScanTasks (e.g. task reads/opens fail, thread pool rejected, IO error) during planFiles.","commonSituations":"Object-store errors while accessing data files during planning, executor thread interruption, or bugs in task generation causing the filling thread to die.","solutions":["Inspect the cause chained to this RuntimeException for the root error.","Check object store/network errors for the data files being planned.","Ensure the streaming query is not being stopped concurrently with planning.","Restart the query; if reproducible, file/inspect Iceberg issues around the async planner thread pool."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  query.awaitTermination();\n} catch (StreamingQueryException e) {\n  if (e.getCause() != null && String.valueOf(e.getCause().getMessage()).contains(\"Queue filling failed\")) {\n    // inspect the chained cause; restart the stream after resolving IO errors\n    restartQueryWithBackoff();\n  } else {\n    throw e;\n  }\n}","preventionTips":["Verify object-store/network stability for data file access.","Avoid stopping the query while a batch is planning.","Alert on this error's chained cause to catch IO regressions early."],"tags":["spark","streaming","queue","threading"],"backgroundTag":"internal-invariant-violation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}