hashicorp/terraform · error

refresh Ecs sts token err, fail to get AccessKeyId: %s

Error message

refresh Ecs sts token err, fail to get AccessKeyId: %s

What it means

Returned by getAuthCredentialByEcsRoleName when jmespath.Search("AccessKeyId", data) fails after Code == "Success". The credential payload is present but the AccessKeyId field could not be evaluated — JMESPath error, not merely a missing key (a missing key would return nil and be caught by the later nil check at line 704).

Source

Thrown at internal/backend/remote-state/oss/backend.go:690

	}
	var data interface{}
	err = json.Unmarshal(response.GetHttpContentBytes(), &data)
	if err != nil {
		err = fmt.Errorf("refresh Ecs sts token err, json.Unmarshal fail: %s", err.Error())
		return
	}
	code, err := jmespath.Search("Code", data)
	if err != nil {
		err = fmt.Errorf("refresh Ecs sts token err, fail to get Code: %s", err.Error())
		return
	}
	if code.(string) != "Success" {
		err = fmt.Errorf("refresh Ecs sts token err, Code is not Success")
		return
	}
	accessKeyId, err := jmespath.Search("AccessKeyId", data)
	if err != nil {
		err = fmt.Errorf("refresh Ecs sts token err, fail to get AccessKeyId: %s", err.Error())
		return
	}
	accessKeySecret, err := jmespath.Search("AccessKeySecret", data)
	if err != nil {
		err = fmt.Errorf("refresh Ecs sts token err, fail to get AccessKeySecret: %s", err.Error())
		return
	}
	securityToken, err := jmespath.Search("SecurityToken", data)
	if err != nil {
		err = fmt.Errorf("refresh Ecs sts token err, fail to get SecurityToken: %s", err.Error())
		return
	}

	if accessKeyId == nil || accessKeySecret == nil || securityToken == nil {
		err = fmt.Errorf("there is no any available accesskey, secret and security token for Ecs role %s", ecsRoleName)
		return
	}

View on GitHub (pinned to c9def3e214)

Solutions

  1. curl the metadata URL and confirm "AccessKeyId" is a top-level string.
  2. Retry — this is almost always transient.
  3. Re-attach the RAM role or recreate it to force a clean metadata envelope.
  4. Fall back to static/STS credentials if it persists.

Example fix

# before: metadata returns Success but no/partial AccessKeyId
ecs_role_name = "partial-role"

# after: re-attach a healthy RAM role whose metadata returns
# {"Code":"Success","AccessKeyId":"STS...",...}
ecs_role_name = "healthy-role"
Defensive patterns

Strategy: validation

Validate before calling

func metadataHasAccessKeyId(role string) error {
    u := "http://100.100.100.200/latest/meta-data/ram/security-credentials/" + role
    resp, err := http.Get(u)
    if err != nil { return err }
    defer resp.Body.Close()
    var m map[string]interface{}
    json.NewDecoder(resp.Body).Decode(&m)
    if _, ok := m["AccessKeyId"].(string); !ok {
        return fmt.Errorf("metadata missing string AccessKeyId: %#v", m)
    }
    return nil
}

Try / catch

if _, err := getAuthCredentialByEcsRoleName(role); err != nil {
    if strings.Contains(err.Error(), "fail to get AccessKeyId") {
        time.Sleep(2 * time.Second)
        return getAuthCredentialByEcsRoleName(role) // bounded retry for partial envelope
    }
    return err
}

Prevention

When it happens

Trigger: The metadata JSON declares Code = Success but the AccessKeyId field has an unexpected type or the document structure breaks JMESPath traversal (e.g. AccessKeyId is a nested object or array rather than a string). Rare; usually a malformed/partial metadata response.

Common situations: Transient metadata service returning a partial/oddly-shaped success envelope; very rare schema drift; intermediary mutating the JSON.

Related errors

AI-assisted analysis of hashicorp/terraform@c9def3e214 (2026-08-07). Data as JSON: /api/errors/dd5a5aa88c78ff3e. Report an issue: GitHub.