elastic/elasticsearch · error · IllegalStateException

STS returned credentials that are already expired at

Error message

STS returned credentials that are already expired at [${expiry}] (now [${now}])

What it means

toCached computes now = clock.instant() and checks expiry.isAfter(now). If the returned credentials already expired at or before now, it throws IllegalStateException with both timestamps. This protects against accepting credentials STS should not have issued.

Solutions

Verify NTP / clock sync on the host running the provider
If in tests, ensure the stubbed expiry is in the future relative to the injected clock
Increase the requested role session duration if STS is returning very short-lived credentials
If clock skew is the cause, correct the system clock and restart the provider

Example fix

// before (test)
Instant now = Instant.now();
Credentials c = Credentials.builder().accessKeyId("k").secretAccessKey("s").expiration(now.minusSeconds(1)).build();
// after
Credentials c = Credentials.builder().accessKeyId("k").secretAccessKey("s").expiration(now.plusSeconds(900)).build();

Defensive patterns

Strategy: validation

Validate before calling

Instant now = clock.instant();
if (!expiry.isAfter(now)) {
    throw new IllegalStateException("credentials expired: " + expiry);
}

Type guard

static boolean isFutureExpiry(Instant expiry, Clock clock) {
    return expiry != null && expiry.isAfter(clock.instant());
}

Try / catch

try { toCached(resp); }
catch (IllegalStateException e) { /* check clock skew, then retry */ }

Prevention

Keep NTP synchronised on hosts running the provider
In tests, stub expiry relative to the injected clock
Treat returned-expired as a clock-skew signal first

When it happens

Trigger: STS returns credentials whose expiration Instant is <= the provider's clock now. The check is strict: expiry must be strictly after now.

Common situations: Clock skew between the ES node and STS (node clock ahead); STS returned short-lived creds that expired in transit; a misconfigured clock source (e.g. wrong NTP); tests with a fixed clock that is already past the stubbed expiry; refresh attempts racing after the cached value already lapsed.

Related errors

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/f57ab8dee1f4e1a2. Report an issue: GitHub.

Appendix: source

Thrown at libs/workload-identity-aws/src/main/java/org/elasticsearch/workload/identity/aws/AsyncWebIdentityCredentialsProvider.java:206

            .roleArn(roleArn)
            .roleSessionName(roleSessionName)
            .webIdentityToken(token)
            .build();
        return stsAsyncClient.assumeRoleWithWebIdentity(request);
    }

    private Cached toCached(AssumeRoleWithWebIdentityResponse response) {
        Credentials credentials = response.credentials();
        if (credentials == null) {
            throw new IllegalStateException("STS AssumeRoleWithWebIdentity response did not include credentials");
        }
        Instant expiry = credentials.expiration();
        if (expiry == null) {
            throw new IllegalStateException("STS AssumeRoleWithWebIdentity response did not include a credential expiry");
        }
        Instant now = clock.instant();
        if (expiry.isAfter(now) == false) {
            throw new IllegalStateException("STS returned credentials that are already expired at [" + expiry + "] (now [" + now + "])");
        }
        AwsSessionCredentials sessionCredentials = AwsSessionCredentials.builder()
            .accessKeyId(credentials.accessKeyId())
            .secretAccessKey(credentials.secretAccessKey())
            .sessionToken(credentials.sessionToken())
            .expirationTime(expiry)
            .build();
        return new Cached(sessionCredentials, expiry.minus(prefetchTime), expiry.minus(staleTime));
    }

    private static Throwable unwrap(Throwable t) {
        return t instanceof CompletionException && t.getCause() != null ? t.getCause() : t;
    }

    /**
     * No-op: the {@link StsAsyncClient} is owned and closed by the caller.
     */
    @Override

View on GitHub (pinned to db6a809a66)