risingwavelabs/risingwave · error

Vault API returned error status: {} - {}

Error message

Vault API returned error status: {} - {}

What it means

The Vault HTTP API responded with a non-2xx status during a secret read. The error embeds the HTTP status code and the raw response body, so it captures any server-side rejection (auth failure, permission denied, malformed path, server error). It is thrown after the token-refresh retry logic decides the response is final.

Source

Thrown at src/common/secret/src/vault_client.rs:214

                .client
                .get(url.as_str())
                .header("X-Vault-Token", &token)
                .send()
                .await
                .context("Failed to send request to Vault")?;

            // Handle authentication failures - token may have been rotated/revoked
            if (response.status() == 401 || response.status() == 403)
                && retry_count == 0
                && matches!(self.config.auth, HashiCorpVaultAuth::AppRole { .. })
            {
                // this case means the token changed during cache, need to trigger a refresh
                force_refresh_token = true;
                continue;
            }

            if !response.status().is_success() {
                return Err(anyhow::anyhow!(
                    "Vault API returned error status: {} - {}",
                    response.status(),
                    response.text().await.unwrap_or_default()
                ));
            }

            // Success case - process the response and break out of retry loop
            return self.process_secret_response(response).await;
        }

        // todo: refine error message
        Err(anyhow::anyhow!("Failed to get secret from Vault"))
    }

    async fn process_secret_response(&self, response: reqwest::Response) -> Result<Vec<u8>> {
        // https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2/cookbook/read-data
        // a demo response:
        //   {

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the status and body in the error message to identify the cause (404 = wrong path, 403 = permissions, 503 = sealed/unavailable)
2. Verify the secret path and KV mount point match the actual Vault layout (KV-v2 paths need data/ prefix handled by the client)
3. Check the token's policies grant 'read' on the secret path via 'vault policy read'
4. If 503, unseal Vault or check cluster health

// Check Vault health before reading secrets
let healthy = reqwest::get(format!("{}/v1/sys/health", vault_addr)).await?.status().is_success();
if !healthy { /* unseal or fix Vault first */ }

match client.get_secret().await {
    Err(e) if e.to_string().contains("403") => fix_policy_and_retry(),
    Err(e) if e.to_string().contains("503") => unseal_vault_and_retry(),
    Err(e) => return Err(e),
    Ok(v) => Ok(v),
}

When it happens

Trigger: get_secret issues a KV-v2 read request and the response status is not success (e.g. 403 ACL denial, 404 unknown path/mount, 503 Vault sealed/standby).

Common situations: Wrong secret path or mount point; Vault policy does not grant read on the path; Vault is sealed or unreachable backend; token lacks capabilities after policy change.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Failed to get secret from Vault
• request error
• schema registry client error: {0}
• confluent registry send req error: {0}
• sending stream load request failed

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/fa123e737861e3fa. Report an issue: GitHub.