{"record":{"id":"397a393bd4e40dca","repo":"temporalio/temporal","slug":"failed-to-retrieve-aws-credentials-w","errorCode":null,"errorMessage":"failed to retrieve AWS credentials: %w","messagePattern":"failed to retrieve AWS credentials: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"common/persistence/visibility/store/elasticsearch/client/aws.go","lineNumber":31,"sourceCode":"\n\t\"github.com/aws/aws-sdk-go-v2/aws\"\n\tv4signer \"github.com/aws/aws-sdk-go-v2/aws/signer/v4\"\n\tawsconfig \"github.com/aws/aws-sdk-go-v2/config\"\n\t\"github.com/aws/aws-sdk-go-v2/credentials\"\n)\n\ntype awsSigningTransport struct {\n\tcreds   aws.CredentialsProvider\n\tsigner  *v4signer.Signer\n\tregion  string\n\tservice string\n\twrapped http.RoundTripper\n}\n\nfunc (t *awsSigningTransport) RoundTrip(req *http.Request) (*http.Response, error) {\n\tcreds, err := t.creds.Retrieve(req.Context())\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to retrieve AWS credentials: %w\", err)\n\t}\n\n\tvar bodyBytes []byte\n\tif req.Body != nil {\n\t\tbodyBytes, err = io.ReadAll(req.Body)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to read request body: %w\", err)\n\t\t}\n\t\treq.Body = io.NopCloser(bytes.NewReader(bodyBytes))\n\t}\n\n\thash := fmt.Sprintf(\"%x\", sha256.Sum256(bodyBytes))\n\terr = t.signer.SignHTTP(req.Context(), creds, req, hash, t.service, t.region, time.Now())\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to sign request: %w\", err)\n\t}\n\n\tif bodyBytes != nil {","sourceCodeStart":13,"sourceCodeEnd":49,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/common/persistence/visibility/store/elasticsearch/client/aws.go#L13-L49","documentation":"The awsSigningTransport used by the Elasticsearch visibility store signs every outgoing HTTP request with AWS SigV4. Before signing it calls CredentialsProvider.Retrieve(ctx); if AWS credential resolution fails (expired, missing, or unrefreshable credentials), RoundTrip returns this error instead of issuing the request.","triggerScenarios":"Any Elasticsearch visibility request routed through NewAwsHttpClient with request signing enabled when the configured credential provider fails: static credentials empty/invalid, 'environment' provider missing AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, 'aws-sdk-default' provider failing to refresh (expired session token, unreachable IMDS/STSI, missing profile).","commonSituations":"EKS/EC2 instance profile or IRSA token not available to the Temporal pod; long-running process with expired session credentials; typo'd AWS_SHARED_CREDENTIALS_FILE or profile name; disabled metadata endpoint in hardened clusters; static keys rotated out from under the config.","solutions":["Verify AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN (or the static visibility config block) are present and valid","Switch credentialProvider to 'aws-sdk-default' so the SDK can auto-refresh via instance profile/IRSA/web identity","Check the underlying wrapped error (e.g. ExpiredToken, NoCredentialProviders) and re-authenticate or rotate the referenced secret","Confirm the pod/network can reach the credential source (IMDS at 169.254.169.254, STS AssumeRoleWithWebIdentity, EKS pod-identity agent)"],"exampleFix":"// before (visibility config with static creds that expired)\ncredentialProvider: static\n// after: let the SDK refresh credentials\ncredentialProvider: aws-sdk-default","handlingStrategy":"retry","validationCode":"// before startup, verify credentials resolve\ncreds, err := cfg.Credentials.Retrieve(context.Background())\nif err != nil {\n    return fmt.Errorf(\"visibility AWS credentials unavailable at startup: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"resp, err := esClient.Search(req)\nif err != nil && strings.Contains(err.Error(), \"failed to retrieve AWS credentials\") {\n    // refresh creds / fail fast, retry once\n    ...\n}","preventionTips":["Prefer credentialProvider 'aws-sdk-default' for automatic refresh over static keys","Ensure IRSA / instance profiles are configured on the Temporal pods","Monitor AWS credential expiry metrics/logs in the visibility path","Validate AWS_REGION and credential env vars in the deployment spec"],"tags":["go","aws","elasticsearch","credentials","sigv4","visibility-store"],"backgroundTag":"aws-credentials-not-found","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}