apache/iceberg · warning

Failed to refresh storage credentials

Error message

Failed to refresh storage credentials

What it means

GCSFileIO.refreshStorageCredentials periodically refreshes delegated/stored storage credentials. When the refresh throws, it logs this warning (with the cause) and keeps the old credentials; the executor is only rescheduled if refresh succeeded and the resource is still open. This can silently lead to expired credentials later.

Solutions

1. Inspect the logged exception cause to fix the underlying token/IAM issue
2. Ensure the workload has rights to the credential/impersonation chain being refreshed
3. Verify network access to the token endpoint from the job environment
4. Restart the job if credentials have fully expired, since refresh failures are swallowed

// pre-check token validity before long jobs
assert tokenExpiry > Instant.now().plus(Duration.ofHours(1));

When it happens

Trigger: The scheduled credential refresh task runs while the identity/credential provider fails (expired OAuth token, network error to the token endpoint, IAM permission changes) and throws an Exception.

Common situations: Long-running jobs where short-lived tokens expire mid-run; temporary network outage to GCS metadata/token server; misconfigured impersonation/service-account permissions introduced after job start.

Related errors

• Failed to refresh storage credentials
• Cannot assume role to sign REST requests because is not…
• Creating BigQuery client failed due to a security issue
• Failed to close the OAuth2RefreshCredentialsHandler
• Failed to create GCP cloud KMS service client

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/ad0069c50d9f80c1. Report an issue: GitHub.

Appendix: source

Thrown at gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java:258

  private void refreshStorageCredentials() {
    if (isResourceClosed.get()) {
      return;
    }

    try (OAuth2RefreshCredentialsHandler handler =
        OAuth2RefreshCredentialsHandler.create(properties)) {
      List<StorageCredential> refreshed =
          handler.fetchCredentials().credentials().stream()
              .filter(c -> c.prefix().startsWith(ROOT_STORAGE_PREFIX))
              .map(c -> StorageCredential.create(c.prefix(), c.config()))
              .toList();

      if (!refreshed.isEmpty() && !isResourceClosed.get()) {
        this.storageCredentials = Lists.newArrayList(refreshed);
        scheduleCredentialRefresh();
      }
    } catch (Exception e) {
      LOG.warn("Failed to refresh storage credentials", e);
    }
  }

  private ScheduledExecutorService executorService() {
    if (executorService == null) {
      synchronized (GCSFileIO.class) {
        if (executorService == null) {
          executorService =
              ThreadPools.newExitingScheduledPool(
                  "iceberg-gcsfileio-tasks", 1, Duration.ofSeconds(10));
        }
      }
    }

    return executorService;
  }

  @Override

View on GitHub (pinned to 86d9c8fc54)