{"record":{"id":"50ee2a8d3d049fe5","repo":"risingwavelabs/risingwave","slug":"failed-to-generate-aws-msk-iam-token","errorCode":null,"errorMessage":"failed to generate AWS MSK IAM token","messagePattern":"failed to generate AWS MSK IAM token","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/connector/src/source/kafka/client_context.rs","lineNumber":150,"sourceCode":"            let region = region.clone();\n            let credentials_provider = credentials_provider.clone();\n            let signer_timeout_sec = *signer_timeout_sec;\n            let (token, expiration_time_ms) = {\n                let result = tokio::task::block_in_place(move || {\n                    KAFKA_SOURCE_RUNTIME.block_on(async {\n                        timeout(\n                            Duration::from_secs(signer_timeout_sec),\n                            generate_auth_token_from_credentials_provider(\n                                region,\n                                credentials_provider,\n                            ),\n                        )\n                        .await\n                    })\n                });\n                result\n                    .map_err(|_e| \"generating AWS MSK IAM token timeout\".to_owned())?\n                    .map_err(|e| anyhow!(e))\n                    .context(\"failed to generate AWS MSK IAM token\")?\n            };\n            Ok(OAuthToken {\n                token,\n                principal_name: \"\".to_owned(),\n                lifetime_ms: expiration_time_ms,\n            })\n        } else {\n            Err(\"must provide AWS IAM credential\".into())\n        }\n    }\n\n    fn enable_refresh_oauth_token(&self) -> bool {\n        self.auth.is_some()\n    }\n}\n\npub type BoxConsumerContext = Box<dyn ConsumerContext>;","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/source/kafka/client_context.rs#L132-L168","documentation":"This wraps any failure that occurs while generating the short-lived MSK IAM auth token used for SASL/OAUTHBEARER, including the signer timeout (`generating AWS MSK IAM token timeout`). The token generation call to aws_msk_iam_sasl_signer either errored or exceeded the configured signer timeout.","triggerScenarios":"During kafka client authentication (`generate_oauth_token` callback) when `generate_auth_token_from_credentials_provider` fails (invalid/expired credentials, network issues reaching the signer endpoint) or exceeds `msk_signer_timeout_sec` (default 10s).","commonSituations":"Expired or revoked IAM credentials; instance metadata service (IMDSv2) unreachable or hops limit too low in containers; VPC without route to AWS endpoints; slow credential provider causing the 10s timeout; wrong region preventing endpoint resolution.","solutions":["Increase `aws.auth.msk_signer_timeout_sec` in the WITH options if the failure is a timeout","Verify credentials are valid and not expired (`aws sts get-caller-identity`)","Check network connectivity to AWS endpoints from the compute node (IMDS at 169.254.169.254, STS); in containers/ENI-based pods raise IMDSv2 hop limit","Confirm `aws.region` matches the MSK cluster region"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// pre-check connectivity/creds\nlet ok = std::process::Command::new(\"aws\").args([\"sts\",\"get-caller-identity\"]).status().map(|s| s.success()).unwrap_or(false);","typeGuard":null,"tryCatchPattern":"match err {\n    e if e.to_string().contains(\"timeout\") => increase msk_signer_timeout_sec and retry,\n    e => fix credentials/network before retrying,\n}","preventionTips":["Raise aws.auth.msk_signer_timeout_sec in slow networks","Ensure IMDS reachable with adequate hop limit in containerized deployments","Rotate credentials before expiry and monitor for auth failures"],"tags":["kafka","aws","msk","iam","timeout","network"],"backgroundTag":"request-timeout","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}