apache/iceberg · critical · RuntimeException

Table refresh failed

Error message

Table refresh failed

What it means

The async planner refreshes the Iceberg table on a background thread; failures there are captured in refreshFailedThrowable rather than thrown immediately. When the main planFiles loop exits, it rethrows the captured cause wrapped in a RuntimeException('Table refresh failed'). This surfaces table refresh problems (e.g. metadata read errors) at the point Spark expects batch planning to complete.

Solutions

1. Inspect the wrapped cause (getCause) for the real failure — storage, catalog, or metadata error.
2. Verify underlying storage/catalog connectivity and credentials from the Spark environment.
3. Check that snapshot expiration hasn't removed metadata files the stream still needs; raise retention or checkpoint more often.
4. Restart the streaming query from checkpoint after fixing the root cause.

try {
    query.processAllAvailable();
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().equals("Table refresh failed")) {
        Throwable cause = e.getCause(); // storage/catalog error — alert and restart from checkpoint
    }
}

When it happens

Trigger: The background refresh thread fails (FileIO errors, corrupted/expired metadata, permission failures reading table metadata, REST catalog HTTP errors) while planFiles is running; the stored throwable is then rethrown when the planning loop terminates.

Common situations: Metadata JSON expired/deleted by concurrent expiration or retention misconfiguration; lost access to object storage credentials mid-stream; catalog outage (REST/HC) during streaming; concurrent table replacement (drop/recreate) invalidating the loaded table.

Related errors

• Table refresh failed
• Table refresh failed
• Table refresh failed
• Cannot load current offset at snapshot
• Cannot load current offset at snapshot

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/4d2a223f58d1ab7a. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:216

              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());
              }
              // end offset may be synthetic and not exist in the queue
              boolean endOffsetSynthetic =
                  currentOffset.snapshotId() == endOffset.snapshotId()
                      && (currentOffset.position() + 1) == endOffset.position();
              shouldTerminate = endOffsetPeek || endOffsetSynthetic;
            } else {
              LOG.trace("planFiles hasn't reached {}, waiting", endOffset);
            }
          } while (!shouldTerminate
              && refreshFailedThrowable == null
              && fillQueueFailedThrowable == null);

          if (refreshFailedThrowable != null) {
            throw new RuntimeException("Table refresh failed", refreshFailedThrowable);
          }

          if (fillQueueFailedThrowable != null) {
            throw new RuntimeException("Queue filling failed", fillQueueFailedThrowable);
          }

          LOG.info(
              "completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}",
              table().name(),
              startOffset,
              endOffset,
              filesInPlan,
              rowsInPlan);
          return result;
        });
  }

  /**

View on GitHub (pinned to 86d9c8fc54)