{"record":{"id":"70da8fe462251e56","repo":"risingwavelabs/risingwave","slug":"failed-to-get-secret-from-vault","errorCode":null,"errorMessage":"Failed to get secret from Vault","messagePattern":"Failed to get secret from Vault","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/common/secret/src/vault_client.rs","lineNumber":226,"sourceCode":"                // this case means the token changed during cache, need to trigger a refresh\n                force_refresh_token = true;\n                continue;\n            }\n\n            if !response.status().is_success() {\n                return Err(anyhow::anyhow!(\n                    \"Vault API returned error status: {} - {}\",\n                    response.status(),\n                    response.text().await.unwrap_or_default()\n                ));\n            }\n\n            // Success case - process the response and break out of retry loop\n            return self.process_secret_response(response).await;\n        }\n\n        // todo: refine error message\n        Err(anyhow::anyhow!(\"Failed to get secret from Vault\"))\n    }\n\n    async fn process_secret_response(&self, response: reqwest::Response) -> Result<Vec<u8>> {\n        // https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2/cookbook/read-data\n        // a demo response:\n        //   {\n        //     \"request_id\": \"e345b77b-8b5a-552b-eb2c-7d80a627c9ad\",\n        //     \"lease_id\": \"\",\n        //     \"renewable\": false,\n        //     \"lease_duration\": 0,\n        //     \"data\": {\n        //       \"data\": {\n        //         \"key\": \"test-api-key-12345\",\n        //         \"secret\": \"test-api-secret-67890\"\n        //       },\n        //       \"metadata\": {\n        //         \"created_time\": \"2025-07-17T08:07:24.177261949Z\",\n        //         \"custom_metadata\": null,","sourceCodeStart":208,"sourceCodeEnd":244,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/common/secret/src/vault_client.rs#L208-L244","documentation":"A catch-all failure returned by get_secret after exhausting its retry loop: every attempt (including forced token refreshes) failed to produce a secret. The message is admittedly unrefined (there is a todo to improve it), so the real cause must be found in earlier logs from the individual attempts.","triggerScenarios":"get_secret retries the Vault read request the configured number of times; on token-mismatch it forces a token refresh, but after the final iteration no successful response was obtained and this error is returned.","commonSituations":"Vault persistently returning errors (auth failing on every refreshed token, network flakiness across all retries); concurrent token invalidation racing with reads; Vault outage during the whole retry window.","solutions":["Check logs immediately preceding this error for the underlying per-attempt errors (login failures, HTTP statuses)","Verify Vault address, role_id/secret_id are correct and the AppRole is not revoked","Increase retry tolerance or fix network connectivity to Vault","Test manually with 'vault kv get' using the same token/credentials to isolate the cause"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-flight: verify credentials work before dependent operations\nvault_client.get_token_internal().await?; // fails fast with a specific error","typeGuard":null,"tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"Failed to get secret from Vault\") => {\n        // inspect earlier logs for per-attempt cause, then retry with backoff\n        retry_with_backoff().await\n    }\n    other => other,\n}","preventionTips":["Enable debug logging around Vault reads to capture per-attempt errors","Avoid concurrently rotating Vault tokens while reads are in flight","Alert on Vault availability so prolonged outages don't exhaust retries"],"tags":["vault","retry","network"],"backgroundTag":"api-request-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}