What causes "Failed to get secret from Vault" in risingwavelabs/risingwave?

Check logs immediately preceding this error for the underlying per-attempt errors (login failures, HTTP statuses)

Failed to get secret from Vault

Error message

Failed to get secret from Vault

What it means

A catch-all failure returned by get_secret after exhausting its retry loop: every attempt (including forced token refreshes) failed to produce a secret. The message is admittedly unrefined (there is a todo to improve it), so the real cause must be found in earlier logs from the individual attempts.

Source

Thrown at src/common/secret/src/vault_client.rs:226

                // this case means the token changed during cache, need to trigger a refresh
                force_refresh_token = true;
                continue;
            }

            if !response.status().is_success() {
                return Err(anyhow::anyhow!(
                    "Vault API returned error status: {} - {}",
                    response.status(),
                    response.text().await.unwrap_or_default()
                ));
            }

            // Success case - process the response and break out of retry loop
            return self.process_secret_response(response).await;
        }

        // todo: refine error message
        Err(anyhow::anyhow!("Failed to get secret from Vault"))
    }

    async fn process_secret_response(&self, response: reqwest::Response) -> Result<Vec<u8>> {
        // https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2/cookbook/read-data
        // a demo response:
        //   {
        //     "request_id": "e345b77b-8b5a-552b-eb2c-7d80a627c9ad",
        //     "lease_id": "",
        //     "renewable": false,
        //     "lease_duration": 0,
        //     "data": {
        //       "data": {
        //         "key": "test-api-key-12345",
        //         "secret": "test-api-secret-67890"
        //       },
        //       "metadata": {
        //         "created_time": "2025-07-17T08:07:24.177261949Z",
        //         "custom_metadata": null,

View on GitHub (pinned to 6469eb736d)

Solutions

Check logs immediately preceding this error for the underlying per-attempt errors (login failures, HTTP statuses)
Verify Vault address, role_id/secret_id are correct and the AppRole is not revoked
Increase retry tolerance or fix network connectivity to Vault
Test manually with 'vault kv get' using the same token/credentials to isolate the cause

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: verify credentials work before dependent operations
vault_client.get_token_internal().await?; // fails fast with a specific error

Try / catch

match result {
    Err(e) if e.to_string().contains("Failed to get secret from Vault") => {
        // inspect earlier logs for per-attempt cause, then retry with backoff
        retry_with_backoff().await
    }
    other => other,
}

Prevention

Enable debug logging around Vault reads to capture per-attempt errors
Avoid concurrently rotating Vault tokens while reads are in flight
Alert on Vault availability so prolonged outages don't exhaust retries

When it happens

Trigger: get_secret retries the Vault read request the configured number of times; on token-mismatch it forces a token refresh, but after the final iteration no successful response was obtained and this error is returned.

Common situations: Vault persistently returning errors (auth failing on every refreshed token, network flakiness across all retries); concurrent token invalidation racing with reads; Vault outage during the whole retry window.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/70da8fe462251e56. Report an issue: GitHub.