apache/pulsar · error · AuthenticationException

INVALID_TOKEN

INVALID_TOKEN

Error message

Athenz Role Token Not Authenticated from Client: %s

What it means

AuthenticationProviderAthenz validates the Athenz role token (ZTS-issued, signed ZTS header) presented by a client. When the token's signature or claims fail verification against the trusted Athenz domain/public key, the provider returns a non-authenticated result and throws this AuthenticationException with ErrorCode.INVALID_TOKEN. The message includes only the client address, never the token, to avoid leaking credentials.

Source

Thrown at pulsar-broker-auth-athenz/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderAthenz.java:166

            // Synchronize for non-thread safe static calls inside athenz library
            synchronized (this) {
                PublicKey ztsPublicKey = AuthZpeClient.getZtsPublicKey(token.getKeyId());

                if (ztsPublicKey == null) {
                    errorCode = ErrorCode.NO_PUBLIC_KEY;
                    throw new AuthenticationException("Unable to retrieve ZTS Public Key");
                }

                if (token.validate(ztsPublicKey, allowedOffset, false, null)) {
                    log.debug().attr("roleToken", roleToken)
                            .attr("clientAddress", clientAddress)
                            .log("Athenz Role Token Authenticated for Client");
                    authenticationMetrics.recordSuccess();
                    return token.getPrincipal();
                } else {
                    errorCode = ErrorCode.INVALID_TOKEN;
                    throw new AuthenticationException(
                            String.format("Athenz Role Token Not Authenticated from Client: %s", clientAddress));
                }
            }
        } catch (AuthenticationException exception) {
            incrementFailureMetric(errorCode);
            throw exception;
        }
    }

    @Override
    public void close() throws IOException {
    }

    @VisibleForTesting
    int getAllowedOffset() {
        return this.allowedOffset;
    }
}

View on GitHub (pinned to 820761864e)

Solutions

1. Regenerate/refresh the client's Athenz role token (it is short-lived; ensure the client refetches from ZTS before expiry)
2. Verify the client's principal/role is defined in the correct Athenz domain and the broker trusts the matching ZTS public key (check the ZTS public key file configured on the broker)
3. Check clock synchronization (NTP) between client, broker, and ZTS so token validity windows line up
4. Confirm the client sends the token in the expected format: 'Athenz' + space + role-token in the Authorization header, with the role token's signed header version matching the ZTS key version

Example fix

// before (client caches token forever)
String token = loadTokenFromDisk();
request.setHeader("Athenz", token);
// after (refresh before use)
RoleToken token = ztsClient.getRoleToken(domain, null);
if (token.getToken() != null && !isExpired(token)) {
    request.setHeader("Athenz", token.getToken());
}

// client-side: ensure a fresh role token before connecting
RoleToken rt = zts.getRoleToken(domain, null);
Instant expiry = Instant.ofEpochMilli(rt.getExpiryTime() * 1000);
if (Instant.now().isAfter(expiry.minus(Duration.ofMinutes(1)))) {
    rt = zts.getRoleToken(domain, Duration.ofHours(1));
}

boolean isUsableAthenzToken(RoleToken t) {
    return t != null && t.getToken() != null
        && Instant.ofEpochMilli(t.getExpiryTime() * 1000).isAfter(Instant.now());
}

try {
    authData = provider.authenticate(authDataSource);
} catch (AuthenticationException e) {
    if (String.valueOf(e.getMessage()).contains("Not Authenticated")) {
        refreshTokenAndReconnect();
    } else {
        throw e;
    }
}

When it happens

Trigger: A client sends an 'Athenz' authorization header whose role token fails validation in verifyAthenzRoleTokenAndReturnPrincipal (bad/expired token, wrong domain, signature check failure) during authenticate() of an inbound connection.

Common situations: Client configured with an expired or rotated role token; token fetched from the wrong Athenz domain/service; ZTS public key changed on the broker but clients cache old tokens; clock skew between client and broker making token look expired; misconfigured athenz.conf or missing ZTS public key file on the broker.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• Cannot start the service once it was stopped
• webServicePort/webServicePortTls or http/https bindAddresses
• The retention size must > the backlog quota limit size, but
• The retention time must > the backlog quota limit time, but
• brokerDeleteInactiveTopicsEnabled and brokerCloseInactiveTop

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/e7bf445da4da2a7d. Report an issue: GitHub.