kubernetes/kops · critical

getting AWS credentials: %w

Error message

getting AWS credentials: %w

What it means

createTokenV1 wraps any failure from the AWS SDK credentials provider when retrieving credentials for SigV4 signing of the kOps bootstrap GetCallerIdentity request. The credentials chain (env vars, shared config, IMDS, IRSA/EKS pod identity, container credentials) returned an error, so a signed token cannot be minted. This is thrown by kOps' AWS authenticator in pkg/bootstrap/awsbootstrap.

Source

Thrown at pkg/bootstrap/awsbootstrap/authenticator.go:115

	// The issue is that if we upgrade the nodes before the control plane,
	// the nodes are using v2 authentication against a v1 verifier.
	// By having the server support v1 and v2, but the nodes continue to use
	// v1 for now, we can introduce v2 support and then enable it in a few versions.
	// The "nodes before control plane" is not the common case,
	// and nodes at much higher versions is not guaranteed to be supported by kube,
	// so once we are at kOps 1.32 this shoud be safe to flip to use V2.
	// It's possibly safe at kOps 1.31 but that might need more careful analysis.
	signWithV1 := true
	if signWithV1 {
		return a.createTokenV1(ctx, body)
	}
	return a.createTokenV2(ctx, body)
}

func (a *awsAuthenticator) createTokenV1(ctx context.Context, body []byte) (string, error) {
	credentials, err := a.credentialsProvider.Retrieve(ctx)
	if err != nil {
		return "", fmt.Errorf("getting AWS credentials: %w", err)
	}

	host, err := a.getSTSHost(ctx)
	if err != nil {
		return "", fmt.Errorf("getting AWS STS url: %w", err)
	}
	stsURL := "https://" + host + "/"
	region := a.region

	req, err := signV1Request(ctx, stsURL, region, credentials, time.Now(), body)
	if err != nil {
		return "", fmt.Errorf("building (v1) signed request: %w", err)
	}
	headers, err := json.Marshal(req.Header)
	if err != nil {
		return "", fmt.Errorf("converting headers to json: %w", err)
	}
	return AWSAuthenticationTokenPrefixV1 + base64.StdEncoding.EncodeToString(headers), nil

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify credentials resolve in the same environment: run `aws sts get-caller-identity` with the same env/profile/user.
2. Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and (if needed) AWS_SESSION_TOKEN, or configure the correct AWS_PROFILE / AWS_SHARED_CREDENTIALS_FILE.
3. On EKS, attach an IRSA service account annotation (eks.amazonaws.com/role-arn) or use EKS Pod Identity, and confirm the trust policy allows the pod.
4. If relying on IMDS, check the instance has an IAM role attached and that IMDSv2 hop limit >= 2 for containers.
5. Check region configuration (AWS_REGION/AWS_DEFAULT_REGION) and that STS is reachable from the network.

Example fix

// before: authenticator built with no creds source
auth, err := awsbootstrap.NewAuthenticator(ctx, region, nil, nil)
// after: pass an explicit, resolvable provider
creds := aws.NewCredentialsCache(aws.NewStaticCredentialsProvider(key, secret, token))
auth, err := awsbootstrap.NewAuthenticator(ctx, region, stsClient, creds)

creds, err := credProvider.Retrieve(ctx)
if err != nil || creds.AccessKeyID == "" {
	return fmt.Errorf("AWS credentials unavailable: %w", err)
}

token, err := auth.CreateToken(body)
if err != nil {
	var credErr *aws.CredentialsCacheError
	if errors.As(err, &credErr) { /* re-authenticate / refresh creds */ }
}

When it happens

Trigger: a.credentialsProvider.Retrieve(ctx) returns an error during CreateToken (v1 path): no AWS credentials resolvable in the process environment, expired/stale credentials, IMDS unreachable/timing out, or an invalid credentials config passed to awsAuthenticator creation.

Common situations: Running `kops get addons`/bootstrap commands outside a cluster node without AWS_ACCESS_KEY_ID/SECRET set; ~/.aws/credentials missing or malformed; on EKS without an IRSA service-account token or pod identity agent; EC2 IMDSv2 hop limit blocking metadata from a container; expired session tokens with AWS_SESSION_TOKEN stale.

Related errors

• failed to load aws config: %w
• error getting AWS account ID: %v
• DIGITALOCEAN_ACCESS_TOKEN is required
• error loading default AWS config: %v
• node identity is required

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/5925374325750d46. Report an issue: GitHub.