elastic/elasticsearch · error · IllegalStateException

STS returned credentials that are already expired at [${expi

Error message

STS returned credentials that are already expired at [${expiry}] (now [${now}])

What it means

toCached computes now = clock.instant() and checks expiry.isAfter(now). If the returned credentials already expired at or before now, it throws IllegalStateException with both timestamps. This protects against accepting credentials STS should not have issued.

Source

Thrown at libs/workload-identity-aws/src/main/java/org/elasticsearch/workload/identity/aws/AsyncWebIdentityCredentialsProvider.java:206

            .roleArn(roleArn)
            .roleSessionName(roleSessionName)
            .webIdentityToken(token)
            .build();
        return stsAsyncClient.assumeRoleWithWebIdentity(request);
    }

    private Cached toCached(AssumeRoleWithWebIdentityResponse response) {
        Credentials credentials = response.credentials();
        if (credentials == null) {
            throw new IllegalStateException("STS AssumeRoleWithWebIdentity response did not include credentials");
        }
        Instant expiry = credentials.expiration();
        if (expiry == null) {
            throw new IllegalStateException("STS AssumeRoleWithWebIdentity response did not include a credential expiry");
        }
        Instant now = clock.instant();
        if (expiry.isAfter(now) == false) {
            throw new IllegalStateException("STS returned credentials that are already expired at [" + expiry + "] (now [" + now + "])");
        }
        AwsSessionCredentials sessionCredentials = AwsSessionCredentials.builder()
            .accessKeyId(credentials.accessKeyId())
            .secretAccessKey(credentials.secretAccessKey())
            .sessionToken(credentials.sessionToken())
            .expirationTime(expiry)
            .build();
        return new Cached(sessionCredentials, expiry.minus(prefetchTime), expiry.minus(staleTime));
    }

    private static Throwable unwrap(Throwable t) {
        return t instanceof CompletionException && t.getCause() != null ? t.getCause() : t;
    }

    /**
     * No-op: the {@link StsAsyncClient} is owned and closed by the caller.
     */
    @Override

View on GitHub (pinned to db6a809a66)

Solutions

1. Verify NTP / clock sync on the host running the provider
2. If in tests, ensure the stubbed expiry is in the future relative to the injected clock
3. Increase the requested role session duration if STS is returning very short-lived credentials
4. If clock skew is the cause, correct the system clock and restart the provider

Example fix

// before (test)
Instant now = Instant.now();
Credentials c = Credentials.builder().accessKeyId("k").secretAccessKey("s").expiration(now.minusSeconds(1)).build();
// after
Credentials c = Credentials.builder().accessKeyId("k").secretAccessKey("s").expiration(now.plusSeconds(900)).build();

Instant now = clock.instant();
if (!expiry.isAfter(now)) {
    throw new IllegalStateException("credentials expired: " + expiry);
}

static boolean isFutureExpiry(Instant expiry, Clock clock) {
    return expiry != null && expiry.isAfter(clock.instant());
}

try { toCached(resp); }
catch (IllegalStateException e) { /* check clock skew, then retry */ }

When it happens

Trigger: STS returns credentials whose expiration Instant is <= the provider's clock now. The check is strict: expiry must be strictly after now.

Common situations: Clock skew between the ES node and STS (node clock ahead); STS returned short-lived creds that expired in transit; a misconfigured clock source (e.g. wrong NTP); tests with a fixed clock that is already past the stubbed expiry; refresh attempts racing after the cached value already lapsed.

Related errors

• STS AssumeRoleWithWebIdentity response did not include crede
• STS AssumeRoleWithWebIdentity response did not include a cre
• staleTime must be a positive duration but was [${staleTime}]
• prefetchTime must be a positive duration but was [${prefetch
• prefetchTime [${prefetchTime}] must be greater than or equal

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/f57ab8dee1f4e1a2. Report an issue: GitHub.