apache/iceberg · critical · RuntimeException

Table refresh failed

Error message

Table refresh failed

What it means

planFiles runs a background refresh loop whose failures are captured in refreshFailedThrowable and rethrown after the loop ends as "Table refresh failed" with the original exception as cause. It means the table refresh inside the async micro-batch planner failed, so planning cannot produce a valid end offset.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:216

              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();
              boolean endOffsetPeek = false;
              if (nextElem != null) {
                endOffsetPeek = endOffset.equals(nextElem.first());
              }
              // end offset may be synthetic and not exist in the queue
              boolean endOffsetSynthetic =
                  currentOffset.snapshotId() == endOffset.snapshotId()
                      && (currentOffset.position() + 1) == endOffset.position();
              shouldTerminate = endOffsetPeek || endOffsetSynthetic;
            } else {
              LOG.trace("planFiles hasn't reached {}, waiting", endOffset);
            }
          } while (!shouldTerminate
              && refreshFailedThrowable == null
              && fillQueueFailedThrowable == null);

          if (refreshFailedThrowable != null) {
            throw new RuntimeException("Table refresh failed", refreshFailedThrowable);
          }

          if (fillQueueFailedThrowable != null) {
            throw new RuntimeException("Queue filling failed", fillQueueFailedThrowable);
          }

          LOG.info(
              "completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}",
              table().name(),
              startOffset,
              endOffset,
              filesInPlan,
              rowsInPlan);
          return result;
        });
  }

  /**

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the cause of this RuntimeException for the real refresh failure.
2. Check catalog and object-store connectivity/permissions for the table location.
3. Verify metadata files were not removed by concurrent expire_snapshots; adjust expiration retention.
4. Restart the streaming query after fixing the underlying cause; planning will re-plan from the last committed offset.

try {
  query.awaitTermination();
} catch (StreamingQueryException e) {
  Throwable cause = e.getCause();
  if (cause != null && cause.getMessage() != null && cause.getMessage().contains("Table refresh failed")) {
    // inspect cause.getCause() for the underlying refresh failure; restart after fixing
    restartQueryWithBackoff();
  } else {
    throw e;
  }
}

When it happens

Trigger: The background refresh thread's table refresh throws (e.g. metadata file unreadable, catalog/network error, snapshot expiry race) while the batch planner loop is polling.

Common situations: Snapshot expiration deleting metadata mid-stream, catalog connectivity problems, object-store permission errors, or table concurrently overwritten by writers.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Failed writing offset to: ${initialOffsetLocation}
• Failed reading offset from: ${initialOffsetLocation}
• Table refresh failed
• Queue filling failed
• Table refresh failed

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f4c2677bbe947325. Report an issue: GitHub.