kubernetes/kops · error

error decoding EC2 UserData: %v

Error message

error decoding EC2 UserData: %v

What it means

During Instance.Find, kOps fetches the EC2 instance's UserData attribute via DescribeInstanceAttribute and base64-decodes the returned value. If base64.StdEncoding.DecodeString fails, the task cannot build an accurate 'actual' state and Find aborts with this wrapped error. It indicates the value AWS returned for UserData.Value was not valid base64, which normally points to a corrupted or non-base64-encoded UserData payload stored on the instance.

Source

Thrown at upup/pkg/fi/cloudup/awstasks/instance.go:133

		PrivateIPAddress: i.PrivateIpAddress,
		InstanceType:     i.InstanceType,
		ImageID:          i.ImageId,
		Name:             findNameTag(i.Tags),
	}

	// Fetch instance UserData
	{
		request := &ec2.DescribeInstanceAttributeInput{}
		request.InstanceId = i.InstanceId
		request.Attribute = ec2types.InstanceAttributeNameUserData
		response, err := cloud.EC2().DescribeInstanceAttribute(ctx, request)
		if err != nil {
			return nil, fmt.Errorf("error querying EC2 for user metadata for instance %q: %v", *i.InstanceId, err)
		}
		if response.UserData != nil {
			b, err := base64.StdEncoding.DecodeString(aws.ToString(response.UserData.Value))
			if err != nil {
				return nil, fmt.Errorf("error decoding EC2 UserData: %v", err)
			}
			actual.UserData = fi.NewBytesResource(b)
		}
	}

	if i.SubnetId != nil {
		actual.Subnet = &Subnet{ID: i.SubnetId}
	}
	if i.KeyName != nil {
		actual.SSHKey = &SSHKey{Name: i.KeyName}
	}

	for _, sg := range i.SecurityGroups {
		actual.SecurityGroups = append(actual.SecurityGroups, &SecurityGroup{ID: sg.GroupId})
	}

	associatePublicIpAddress := false
	for _, ni := range i.NetworkInterfaces {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Inspect the instance's UserData with `aws ec2 get-instance-attribute --attribute userData --instance-id <id>` and verify it is valid base64 (`echo <value> | base64 -d`).
2. Re-apply the UserData through kOps (delete/recreate the instance or fix the task spec) so AWS stores a properly encoded value.
3. If the instance is externally managed, mark it as Shared so kOps does not diff its UserData.
4. Check for tools that rewrote the attribute and disable them; then rerun `kops update cluster`.
5. Upgrade kOps/SDK if a version bug in attribute decoding is suspected.

Example fix

// before
b, err := base64.StdEncoding.DecodeString(aws.ToString(response.UserData.Value))
if err != nil {
	return nil, fmt.Errorf("error decoding EC2 UserData: %v", err)
}
// after
raw := aws.ToString(response.UserData.Value)
b, err := base64.StdEncoding.DecodeString(raw)
if err != nil {
	// tolerate externally-managed (non-base64) userdata instead of failing Find
	klog.V(2).Infof("ignoring non-base64 UserData for instance: %v", err)
} else {
	actual.UserData = fi.NewBytesResource(b)
}

// Validate base64 decodability before relying on kOps Find / stored UserData
import "encoding/base64"

func isValidBase64(s string) bool {
	_, err := base64.StdEncoding.DecodeString(s)
	return err == nil
}

raw, _ := ec2Client.DescribeInstanceAttribute(ctx, &ec2.DescribeInstanceAttributeInput{
	InstanceId: aws.String(instID), Attribute: ec2types.InstanceAttributeNameUserData,
})
if raw.UserData != nil && !isValidBase64(aws.ToString(raw.UserData.Value)) {
	// fix or exclude this instance before running kops update
}

func hasDecodableUserData(v *string) bool {
	if v == nil {
		return true
	}
	_, err := base64.StdEncoding.DecodeString(*v)
	return err == nil
}

actual, err := instanceTask.Find(ctx, cloud)
if err != nil {
	if strings.Contains(err.Error(), "error decoding EC2 UserData") {
		// treat as externally managed / re-apply userdata
		klog.Warningf("skipping instance with undecodable userdata: %v", err)
		return nil
	}
	return err
}

When it happens

Trigger: Instance.Find on an existing EC2 instance whose DescribeInstanceAttribute(UserData) response contains a UserData.Value string that fails base64 decoding (malformed characters, wrong padding, or a raw/JSON-encoded string stored outside kOps).

Common situations: Instances imported or mutated outside kOps (e.g. created by terraform/CloudFormation/CLI with UserData set differently), UserData edited by automation that wrote plain text or double-encoded data, stale snapshots of instance attributes, or SDK/region mismatches returning unexpected attribute payloads.

Related errors

• error decoding userdata: %s
• error rendering Instance UserData: %v
• Instance UserData was too large (%d bytes)
• DIGITALOCEAN_ACCESS_TOKEN is required
• timed out waiting for volume to detach

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/543c62f181f39c32. Report an issue: GitHub.