apache/iceberg · error · RuntimeException

Table refresh failed

Error message

Table refresh failed

What it means

The async micro-batch planner refreshes the table and fills a planning queue on background tasks, recording any Throwable in refreshFailedThrowable. After the planning loop exits, if a refresh failure was captured, planFiles rethrows it as RuntimeException 'Table refresh failed' with the original cause attached. This surfaces background table-refresh failures to the streaming query.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:216

              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());
              }
              // end offset may be synthetic and not exist in the queue
              boolean endOffsetSynthetic =
                  currentOffset.snapshotId() == endOffset.snapshotId()
                      && (currentOffset.position() + 1) == endOffset.position();
              shouldTerminate = endOffsetPeek || endOffsetSynthetic;
            } else {
              LOG.trace("planFiles hasn't reached {}, waiting", endOffset);
            }
          } while (!shouldTerminate
              && refreshFailedThrowable == null
              && fillQueueFailedThrowable == null);

          if (refreshFailedThrowable != null) {
            throw new RuntimeException("Table refresh failed", refreshFailedThrowable);
          }

          if (fillQueueFailedThrowable != null) {
            throw new RuntimeException("Queue filling failed", fillQueueFailedThrowable);
          }

          LOG.info(
              "completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}",
              table().name(),
              startOffset,
              endOffset,
              filesInPlan,
              rowsInPlan);
          return result;
        });
  }

  /**

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Read the wrapped cause (refreshFailedThrowable) to identify the real failure and fix it (credentials, network, catalog).
2. Add retry/credentials refresh: configure a session token provider or re-login mechanism for the FileIO/catalog.
3. Restart the streaming query once the underlying storage/catalog issue is resolved.
4. Check table metadata health and catalog availability; verify the table hasn't been dropped or made unreadable.

Example fix

// before: long-lived static creds
spark.conf.set("s3.access-key", ...);
// after: refreshable provider
spark.conf.set("s3.session-token-provider.type", "container"); // or IAM role

try { planFiles(...); } catch (RuntimeException e) { if ("Table refresh failed".equals(e.getMessage())) { Throwable cause = e.getCause(); /* check creds/catalog and retry or restart the stream */ } else throw e; }

When it happens

Trigger: During planFiles' loop, the background table refresh task fails (e.g. metadata read error, catalog unreachable, permission revoked, invalid/expired credentials), the loop terminates, and the stored throwable is wrapped and thrown.

Common situations: S3/GCS/Azure credentials expiring mid-stream; catalog (REST/Hive/Hadoop) temporarily unavailable; table metadata corrupted or concurrently rewritten causing refresh errors; network partitions between the driver and object storage.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Table refresh failed
• Table refresh failed
• Table refresh failed
• Interrupted while polling queue
• Queue filling failed

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/7eb1bc4fac97d593. Report an issue: GitHub.