{"record":{"id":"f57ab8dee1f4e1a2","repo":"elastic/elasticsearch","slug":"sts-returned-credentials-that-are-already-expired","errorCode":null,"errorMessage":"STS returned credentials that are already expired at [${expiry}] (now [${now}])","messagePattern":"STS returned credentials that are already expired at \\[(.+?)\\] \\(now \\[(.+?)\\]\\)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"libs/workload-identity-aws/src/main/java/org/elasticsearch/workload/identity/aws/AsyncWebIdentityCredentialsProvider.java","lineNumber":206,"sourceCode":"            .roleArn(roleArn)\n            .roleSessionName(roleSessionName)\n            .webIdentityToken(token)\n            .build();\n        return stsAsyncClient.assumeRoleWithWebIdentity(request);\n    }\n\n    private Cached toCached(AssumeRoleWithWebIdentityResponse response) {\n        Credentials credentials = response.credentials();\n        if (credentials == null) {\n            throw new IllegalStateException(\"STS AssumeRoleWithWebIdentity response did not include credentials\");\n        }\n        Instant expiry = credentials.expiration();\n        if (expiry == null) {\n            throw new IllegalStateException(\"STS AssumeRoleWithWebIdentity response did not include a credential expiry\");\n        }\n        Instant now = clock.instant();\n        if (expiry.isAfter(now) == false) {\n            throw new IllegalStateException(\"STS returned credentials that are already expired at [\" + expiry + \"] (now [\" + now + \"])\");\n        }\n        AwsSessionCredentials sessionCredentials = AwsSessionCredentials.builder()\n            .accessKeyId(credentials.accessKeyId())\n            .secretAccessKey(credentials.secretAccessKey())\n            .sessionToken(credentials.sessionToken())\n            .expirationTime(expiry)\n            .build();\n        return new Cached(sessionCredentials, expiry.minus(prefetchTime), expiry.minus(staleTime));\n    }\n\n    private static Throwable unwrap(Throwable t) {\n        return t instanceof CompletionException && t.getCause() != null ? t.getCause() : t;\n    }\n\n    /**\n     * No-op: the {@link StsAsyncClient} is owned and closed by the caller.\n     */\n    @Override","sourceCodeStart":188,"sourceCodeEnd":224,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/libs/workload-identity-aws/src/main/java/org/elasticsearch/workload/identity/aws/AsyncWebIdentityCredentialsProvider.java#L188-L224","documentation":"toCached computes now = clock.instant() and checks expiry.isAfter(now). If the returned credentials already expired at or before now, it throws IllegalStateException with both timestamps. This protects against accepting credentials STS should not have issued.","triggerScenarios":"STS returns credentials whose expiration Instant is <= the provider's clock now. The check is strict: expiry must be strictly after now.","commonSituations":"Clock skew between the ES node and STS (node clock ahead); STS returned short-lived creds that expired in transit; a misconfigured clock source (e.g. wrong NTP); tests with a fixed clock that is already past the stubbed expiry; refresh attempts racing after the cached value already lapsed.","solutions":["Verify NTP / clock sync on the host running the provider","If in tests, ensure the stubbed expiry is in the future relative to the injected clock","Increase the requested role session duration if STS is returning very short-lived credentials","If clock skew is the cause, correct the system clock and restart the provider"],"exampleFix":"// before (test)\nInstant now = Instant.now();\nCredentials c = Credentials.builder().accessKeyId(\"k\").secretAccessKey(\"s\").expiration(now.minusSeconds(1)).build();\n// after\nCredentials c = Credentials.builder().accessKeyId(\"k\").secretAccessKey(\"s\").expiration(now.plusSeconds(900)).build();","handlingStrategy":"validation","validationCode":"Instant now = clock.instant();\nif (!expiry.isAfter(now)) {\n    throw new IllegalStateException(\"credentials expired: \" + expiry);\n}","typeGuard":"static boolean isFutureExpiry(Instant expiry, Clock clock) {\n    return expiry != null && expiry.isAfter(clock.instant());\n}","tryCatchPattern":"try { toCached(resp); }\ncatch (IllegalStateException e) { /* check clock skew, then retry */ }","preventionTips":["Keep NTP synchronised on hosts running the provider","In tests, stub expiry relative to the injected clock","Treat returned-expired as a clock-skew signal first"],"tags":["aws","sts","credentials","clock-skew","runtime"],"backgroundTag":null,"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}