What causes "failed to refresh Iceberg table credentials: {error:?}" in databendlabs/databend?

Check the debug-formatted inner `error` to see the catalog HTTP status/cause.

failed to refresh Iceberg table credentials

Error message

failed to refresh Iceberg table credentials: {error:?}

What it means

`CatalogCredentialProvider::load` refreshes vended credentials by calling the Iceberg catalog's `load_table`. If that catalog RPC fails for any reason, the failure is wrapped in this anyhow error. The credential provider needs a fresh load_table response to extract vended (temporary) credentials for the storage backend.

Solutions

Check the debug-formatted inner `error` to see the catalog HTTP status/cause.
Verify catalog credentials (catalog auth config) are valid and not expired.
Confirm the table identifier still exists in the catalog (`SHOW TABLES`, or curl the REST catalog).
Check catalog service availability and network reachability from the query node.
Retry — the refresh is retried downstream via `RefreshingAwsCredentialLoader`; a transient catalog blip may resolve itself.

Defensive patterns

Strategy: retry

Try / catch

loop {
    match provider.load().await {
        Ok(c) => break c,
        Err(e) if is_transient(&e) && retries < MAX => { retries += 1; sleep(backoff).await; }
        Err(e) => return Err(e),
    }
}

Prevention

Monitor catalog health and credentials expiry
Keep table identifiers in sync with the catalog (handle external drops/renames)
Alert on repeated refresh failures rather than silently retrying forever

When it happens

Trigger: Periodic or on-demand credential refresh when the Iceberg REST catalog rejects or fails the `load_table` call for `self.table_ident` — auth failures, catalog downtime, 404 on the table identifier, network errors.

Common situations: Expired catalog credentials, REST catalog base URL changed, table dropped/renamed externally in the catalog while Databend still references it, catalog service outage, TLS/DNS problems.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/3dfed1b5fde5f1f1. Report an issue: GitHub.

Appendix: source

Thrown at src/query/storages/iceberg/src/credential.rs:108

#[async_trait]
trait VendedCredentialProvider: Send + Sync {
    async fn load(&self) -> anyhow::Result<VendedCredential>;
}

struct CatalogCredentialProvider {
    catalog: Arc<dyn iceberg::Catalog>,
    table_ident: TableIdent,
}

#[async_trait]
impl VendedCredentialProvider for CatalogCredentialProvider {
    async fn load(&self) -> anyhow::Result<VendedCredential> {
        let table = self
            .catalog
            .load_table(&self.table_ident)
            .await
            .map_err(|error| anyhow!("failed to refresh Iceberg table credentials: {error:?}"))?;
        VendedCredential::from_table(&table).ok_or_else(|| {
            anyhow!("Iceberg load_table response did not contain vended credentials")
        })
    }
}

struct RefreshingAwsCredentialLoader {
    provider: Arc<dyn VendedCredentialProvider>,
    current: Mutex<VendedCredential>,
}

impl RefreshingAwsCredentialLoader {
    fn new(provider: Arc<dyn VendedCredentialProvider>, current: VendedCredential) -> Self {
        Self {
            provider,
            current: Mutex::new(current),
        }
    }

View on GitHub (pinned to 288d84d76e)