temporalio/temporal · error

failed to retrieve AWS credentials: %w

Error message

failed to retrieve AWS credentials: %w

What it means

The awsSigningTransport used by the Elasticsearch visibility store signs every outgoing HTTP request with AWS SigV4. Before signing it calls CredentialsProvider.Retrieve(ctx); if AWS credential resolution fails (expired, missing, or unrefreshable credentials), RoundTrip returns this error instead of issuing the request.

Source

Thrown at common/persistence/visibility/store/elasticsearch/client/aws.go:31

	"github.com/aws/aws-sdk-go-v2/aws"
	v4signer "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
	awsconfig "github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
)

type awsSigningTransport struct {
	creds   aws.CredentialsProvider
	signer  *v4signer.Signer
	region  string
	service string
	wrapped http.RoundTripper
}

func (t *awsSigningTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	creds, err := t.creds.Retrieve(req.Context())
	if err != nil {
		return nil, fmt.Errorf("failed to retrieve AWS credentials: %w", err)
	}

	var bodyBytes []byte
	if req.Body != nil {
		bodyBytes, err = io.ReadAll(req.Body)
		if err != nil {
			return nil, fmt.Errorf("failed to read request body: %w", err)
		}
		req.Body = io.NopCloser(bytes.NewReader(bodyBytes))
	}

	hash := fmt.Sprintf("%x", sha256.Sum256(bodyBytes))
	err = t.signer.SignHTTP(req.Context(), creds, req, hash, t.service, t.region, time.Now())
	if err != nil {
		return nil, fmt.Errorf("failed to sign request: %w", err)
	}

	if bodyBytes != nil {

View on GitHub (pinned to bde624efd1)

Solutions

1. Verify AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN (or the static visibility config block) are present and valid
2. Switch credentialProvider to 'aws-sdk-default' so the SDK can auto-refresh via instance profile/IRSA/web identity
3. Check the underlying wrapped error (e.g. ExpiredToken, NoCredentialProviders) and re-authenticate or rotate the referenced secret
4. Confirm the pod/network can reach the credential source (IMDS at 169.254.169.254, STS AssumeRoleWithWebIdentity, EKS pod-identity agent)

Example fix

// before (visibility config with static creds that expired)
credentialProvider: static
// after: let the SDK refresh credentials
credentialProvider: aws-sdk-default

// before startup, verify credentials resolve
creds, err := cfg.Credentials.Retrieve(context.Background())
if err != nil {
    return fmt.Errorf("visibility AWS credentials unavailable at startup: %w", err)
}

resp, err := esClient.Search(req)
if err != nil && strings.Contains(err.Error(), "failed to retrieve AWS credentials") {
    // refresh creds / fail fast, retry once
    ...
}

When it happens

Trigger: Any Elasticsearch visibility request routed through NewAwsHttpClient with request signing enabled when the configured credential provider fails: static credentials empty/invalid, 'environment' provider missing AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, 'aws-sdk-default' provider failing to refresh (expired session token, unreachable IMDS/STSI, missing profile).

Common situations: EKS/EC2 instance profile or IRSA token not available to the Temporal pod; long-running process with expired session credentials; typo'd AWS_SHARED_CREDENTIALS_FILE or profile name; disabled metadata endpoint in hardened clusters; static keys rotated out from under the config.

Related errors

• failed to read request body: %w
• failed to sign request: %w
• unable to resolve AWS region for obtaining AWS Elastic signi
• unable to create AWS HTTP client for Elasticsearch: %w
• ErrUnableToExecuteActivity

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/397a393bd4e40dca. Report an issue: GitHub.