apache/iceberg · critical · RuntimeException

Table refresh failed

Error message

Table refresh failed

What it means

The async planner refreshes the Iceberg table on a background thread; failures there are captured in refreshFailedThrowable rather than thrown immediately. When the main planFiles loop exits, it rethrows the captured cause wrapped in a RuntimeException('Table refresh failed'). This surfaces table refresh problems (e.g. metadata read errors) at the point Spark expects batch planning to complete.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:216

              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());
              }
              // end offset may be synthetic and not exist in the queue
              boolean endOffsetSynthetic =
                  currentOffset.snapshotId() == endOffset.snapshotId()
                      && (currentOffset.position() + 1) == endOffset.position();
              shouldTerminate = endOffsetPeek || endOffsetSynthetic;
            } else {
              LOG.trace("planFiles hasn't reached {}, waiting", endOffset);
            }
          } while (!shouldTerminate
              && refreshFailedThrowable == null
              && fillQueueFailedThrowable == null);

          if (refreshFailedThrowable != null) {
            throw new RuntimeException("Table refresh failed", refreshFailedThrowable);
          }

          if (fillQueueFailedThrowable != null) {
            throw new RuntimeException("Queue filling failed", fillQueueFailedThrowable);
          }

          LOG.info(
              "completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}",
              table().name(),
              startOffset,
              endOffset,
              filesInPlan,
              rowsInPlan);
          return result;
        });
  }

  /**

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the wrapped cause (getCause) for the real failure — storage, catalog, or metadata error.
2. Verify underlying storage/catalog connectivity and credentials from the Spark environment.
3. Check that snapshot expiration hasn't removed metadata files the stream still needs; raise retention or checkpoint more often.
4. Restart the streaming query from checkpoint after fixing the root cause.

try {
    query.processAllAvailable();
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().equals("Table refresh failed")) {
        Throwable cause = e.getCause(); // storage/catalog error — alert and restart from checkpoint
    }
}

When it happens

Trigger: The background refresh thread fails (FileIO errors, corrupted/expired metadata, permission failures reading table metadata, REST catalog HTTP errors) while planFiles is running; the stored throwable is then rethrown when the planning loop terminates.

Common situations: Metadata JSON expired/deleted by concurrent expiration or retention misconfiguration; lost access to object storage credentials mid-stream; catalog outage (REST/HC) during streaming; concurrent table replacement (drop/recreate) invalidating the loaded table.

Related errors

• Table refresh failed
• Table refresh failed
• Table refresh failed
• Interrupted while polling queue
• Queue filling failed

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/4d2a223f58d1ab7a. Report an issue: GitHub.