apache/iceberg · error · RuntimeException

Queue filling failed

Error message

Queue filling failed

What it means

This wraps any Throwable caught while the background thread fills the planning queue in AsyncSparkMicroBatchPlanner. Queue filling (manifest scanning / file listing for upcoming batches) runs asynchronously; its failure is stored and rethrown as a RuntimeException by planFiles after the background loop terminates.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:220

              }
              // end offset may be synthetic and not exist in the queue
              boolean endOffsetSynthetic =
                  currentOffset.snapshotId() == endOffset.snapshotId()
                      && (currentOffset.position() + 1) == endOffset.position();
              shouldTerminate = endOffsetPeek || endOffsetSynthetic;
            } else {
              LOG.trace("planFiles hasn't reached {}, waiting", endOffset);
            }
          } while (!shouldTerminate
              && refreshFailedThrowable == null
              && fillQueueFailedThrowable == null);

          if (refreshFailedThrowable != null) {
            throw new RuntimeException("Table refresh failed", refreshFailedThrowable);
          }

          if (fillQueueFailedThrowable != null) {
            throw new RuntimeException("Queue filling failed", fillQueueFailedThrowable);
          }

          LOG.info(
              "completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}",
              table().name(),
              startOffset,
              endOffset,
              filesInPlan,
              rowsInPlan);
          return result;
        });
  }

  /**
   * This needs to be non destructive on the queue as spark could call this multiple times. Each
   * time, depending on the table state it could return something different
   *
   * @param startOffset the starting offset of the next microbatch

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check the cause of fillQueueFailedThrowable for the underlying file-listing or manifest error
2. Verify data files are not expired/deleted while the streaming query is running (tune expiring snapshots / history.expire.max-snapshot-age-ms)
3. Ensure object store availability and retry limits
4. Consider disabling the async planner (fall back to sync planning) if concurrent planning races persist

try { planFiles(...) } catch (RuntimeException e) { if (e.getCause() instanceof transient IO error) { retryWithBackoff() } else throw }

When it happens

Trigger: The background fill-queue task throws (e.g. file listing or manifest read error); fillQueueFailedThrowable is non-null when planFiles exits the do/while loop and is rethrown.

Common situations: Data file deletion racing with streaming reads causing missing-file errors; S3/HDFS throttling or outages during file listing; oversized snapshots exhausting memory during queue fill.

Related errors

• Queue filling failed
• Interrupted while polling queue
• Table refresh failed
• Cannot process unknown snapshot operation: ${op.toLowerCase(
• Deleted rows scan task is not supported yet

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d8d221babb64c7ab. Report an issue: GitHub.