apache/iceberg · error · RuntimeException

Table refresh failed

Error message

Table refresh failed

What it means

This error wraps any Throwable caught by the background refresh thread inside AsyncSparkMicroBatchPlanner. The planner refreshes the table asynchronously in a background loop; exceptions from that loop are stored and re-raised as a RuntimeException when the Spark micro-batch thread next calls planFiles, so streaming query failures surface at the batch planning point.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:216

              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());
              }
              // end offset may be synthetic and not exist in the queue
              boolean endOffsetSynthetic =
                  currentOffset.snapshotId() == endOffset.snapshotId()
                      && (currentOffset.position() + 1) == endOffset.position();
              shouldTerminate = endOffsetPeek || endOffsetSynthetic;
            } else {
              LOG.trace("planFiles hasn't reached {}, waiting", endOffset);
            }
          } while (!shouldTerminate
              && refreshFailedThrowable == null
              && fillQueueFailedThrowable == null);

          if (refreshFailedThrowable != null) {
            throw new RuntimeException("Table refresh failed", refreshFailedThrowable);
          }

          if (fillQueueFailedThrowable != null) {
            throw new RuntimeException("Queue filling failed", fillQueueFailedThrowable);
          }

          LOG.info(
              "completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}",
              table().name(),
              startOffset,
              endOffset,
              filesInPlan,
              rowsInPlan);
          return result;
        });
  }

  /**

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Inspect the refreshFailedThrowable cause in the stack trace to identify the underlying refresh failure
  2. Check catalog/metastore connectivity and credentials used by the streaming query
  3. Retry the query; refresh failures are often transient network/metastore issues
  4. Pin table state or reduce refresh frequency if the table is being mutated concurrently by writers
Defensive patterns

Strategy: try-catch

Try / catch

try { planFiles(...) } catch (RuntimeException e) { Throwable cause = e.getCause(); if (cause is transient) restart query with checkpointing; else alert }

Prevention

When it happens

Trigger: A background thread refreshing the Iceberg table (e.g. during ongoing micro-batch planning) throws; the stored refreshFailedThrowable is non-null when planFiles exits its do/while loop and rethrows it.

Common situations: Metastore or file-system outages during streaming reads; catalog authorization failures mid-stream; transient network errors refreshing table metadata from a long-running Spark structured streaming query.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/8b5a3f7b19d43988. Report an issue: GitHub.