What causes "failed to generate AWS MSK IAM token" in risingwavelabs/risingwave?

Increase `aws.auth.msk_signer_timeout_sec` in the WITH options if the failure is a timeout

failed to generate AWS MSK IAM token

Error message

failed to generate AWS MSK IAM token

What it means

This wraps any failure that occurs while generating the short-lived MSK IAM auth token used for SASL/OAUTHBEARER, including the signer timeout (`generating AWS MSK IAM token timeout`). The token generation call to aws_msk_iam_sasl_signer either errored or exceeded the configured signer timeout.

Source

Thrown at src/connector/src/source/kafka/client_context.rs:150

            let region = region.clone();
            let credentials_provider = credentials_provider.clone();
            let signer_timeout_sec = *signer_timeout_sec;
            let (token, expiration_time_ms) = {
                let result = tokio::task::block_in_place(move || {
                    KAFKA_SOURCE_RUNTIME.block_on(async {
                        timeout(
                            Duration::from_secs(signer_timeout_sec),
                            generate_auth_token_from_credentials_provider(
                                region,
                                credentials_provider,
                            ),
                        )
                        .await
                    })
                });
                result
                    .map_err(|_e| "generating AWS MSK IAM token timeout".to_owned())?
                    .map_err(|e| anyhow!(e))
                    .context("failed to generate AWS MSK IAM token")?
            };
            Ok(OAuthToken {
                token,
                principal_name: "".to_owned(),
                lifetime_ms: expiration_time_ms,
            })
        } else {
            Err("must provide AWS IAM credential".into())
        }
    }

    fn enable_refresh_oauth_token(&self) -> bool {
        self.auth.is_some()
    }
}

pub type BoxConsumerContext = Box<dyn ConsumerContext>;

View on GitHub (pinned to 6469eb736d)

Solutions

Increase `aws.auth.msk_signer_timeout_sec` in the WITH options if the failure is a timeout
Verify credentials are valid and not expired (`aws sts get-caller-identity`)
Check network connectivity to AWS endpoints from the compute node (IMDS at 169.254.169.254, STS); in containers/ENI-based pods raise IMDSv2 hop limit
Confirm `aws.region` matches the MSK cluster region

Defensive patterns

Strategy: retry

Validate before calling

// pre-check connectivity/creds
let ok = std::process::Command::new("aws").args(["sts","get-caller-identity"]).status().map(|s| s.success()).unwrap_or(false);

Try / catch

match err {
    e if e.to_string().contains("timeout") => increase msk_signer_timeout_sec and retry,
    e => fix credentials/network before retrying,
}

Prevention

Raise aws.auth.msk_signer_timeout_sec in slow networks
Ensure IMDS reachable with adequate hop limit in containerized deployments
Rotate credentials before expiry and monitor for auth failures

When it happens

Trigger: During kafka client authentication (`generate_oauth_token` callback) when `generate_auth_token_from_credentials_provider` fails (invalid/expired credentials, network issues reaching the signer endpoint) or exceeds `msk_signer_timeout_sec` (default 10s).

Common situations: Expired or revoked IAM credentials; instance metadata service (IMDSv2) unreachable or hops limit too low in containers; VPC without route to AWS endpoints; slow credential provider causing the 10s timeout; wrong region preventing endpoint resolution.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/50ee2a8d3d049fe5. Report an issue: GitHub.