apache/iceberg · error · RuntimeException

Queue filling failed

Error message

Queue filling failed

What it means

In Spark structured streaming reads, AsyncSparkMicroBatchPlanner.planFiles plans batches on a background thread while the main thread runs concurrently. If queue-filling (loading manifests/files into the planning queue) threw, the throwable is captured and re-thrown on the main thread as 'Queue filling failed' with the original cause attached. The actual root cause is in the getCause() chain.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:220

              }
              // end offset may be synthetic and not exist in the queue
              boolean endOffsetSynthetic =
                  currentOffset.snapshotId() == endOffset.snapshotId()
                      && (currentOffset.position() + 1) == endOffset.position();
              shouldTerminate = endOffsetPeek || endOffsetSynthetic;
            } else {
              LOG.trace("planFiles hasn't reached {}, waiting", endOffset);
            }
          } while (!shouldTerminate
              && refreshFailedThrowable == null
              && fillQueueFailedThrowable == null);

          if (refreshFailedThrowable != null) {
            throw new RuntimeException("Table refresh failed", refreshFailedThrowable);
          }

          if (fillQueueFailedThrowable != null) {
            throw new RuntimeException("Queue filling failed", fillQueueFailedThrowable);
          }

          LOG.info(
              "completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}",
              table().name(),
              startOffset,
              endOffset,
              filesInPlan,
              rowsInPlan);
          return result;
        });
  }

  /**
   * This needs to be non destructive on the queue as spark could call this multiple times. Each
   * time, depending on the table state it could return something different
   *
   * @param startOffset the starting offset of the next microbatch

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the chained cause (e.getCause()) with full stack trace to find the real failure (usually IOException from manifest reads).
2. Verify the warehouse/FileIO credentials and network path to the table location are valid for the streaming job.
3. Retry/restart the query; transient object-store errors often resolve on restart.
4. Check that no concurrent table mutation (schema/version rewrite) corrupted the snapshot the planner is reading.

Example fix

// before
// generic wrapper hides root cause; enable cause inspection
try { batch = planFiles(...); } catch (RuntimeException e) {
  log.error("planning failed", e.getCause());
}
// after
// inspect cause; add retry for transient FileIO errors
try { batch = planFiles(...); } catch (RuntimeException e) {
  Throwable root = e.getCause();
  if (isTransient(root)) retryWithBackoff(); else throw e;
}

// pre-check table reachability before starting stream
Table table = catalog.loadTable(ident);
table.refresh(); // throws early if FileIO/credentials are broken

try { planBatch(); } catch (RuntimeException e) {
  if (e.getCause() instanceof IOException && isTransient(e.getCause())) retryWithBackoff();
  else throw e;
}

When it happens

Trigger: Calling SparkStreamingCheckpoints/planFiles when the async planner's queue-filling step throws any exception (typically an IO error reading manifests via FileIO, a table refresh that swapped metadata concurrently, or an unavailable file system) while other errors are absent.

Common situations: Object store transient failures (S3/GCS throttling) during manifest listing; HDFS NameNode unavailability; table deleted or schema changed mid-stream; credentials expiring during long streams.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Table refresh failed
• Failed writing offset to: ${initialOffsetLocation}
• Failed reading offset from: ${initialOffsetLocation}
• Failed to read StreamingOffset from json
• Failed to write StreamingOffset to json

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6c63db43b4e496a3. Report an issue: GitHub.