What causes "Decoding S3A token {cause}" in apache/hadoop?

Put the missing identifier class's jar on the decoding client's classpath

apache/hadoop · error · DelegationTokenIOException

Decoding S3A token {cause}

Error message

Decoding S3A token {cause}

What it means

While decoding a fetched delegation token, S3ADelegationTokens hardens Token.decodeIdentifier: if it throws a RuntimeException whose cause is non-null (typically a ClassNotFoundException or instantiation failure while creating the token identifier class from the token's kind), the cause is wrapped in DelegationTokenIOException 'Decoding S3A token <cause>'. The token payload is fine; the code to materialize its identifier is missing.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADelegationTokens.java:585

   * @param token token to process
   * @return the session token identifier
   * @throws IOException failure to validate/read data encoded in identifier.
   * @throws IllegalArgumentException if the token isn't an S3A session token
   */
  public AbstractS3ATokenIdentifier extractIdentifier(
      final Token<? extends AbstractS3ATokenIdentifier> token)
      throws IOException {

    checkArgument(token != null, "null token");
    AbstractS3ATokenIdentifier identifier;
    // harden up decode beyond that Token does itself
    try {
      identifier = token.decodeIdentifier();
    } catch (RuntimeException e) {
      Throwable cause = e.getCause();
      if (cause != null) {
        // its a wrapping around class instantiation.
        throw new DelegationTokenIOException("Decoding S3A token " + cause,
            cause);
      } else {
        throw e;
      }
    }
    if (identifier == null) {
      throw new DelegationTokenIOException("Failed to unmarshall token for "
          + getCanonicalUri());
    }
    identifier.validate();
    return identifier;
  }

  /**
   * Return a string for use in building up the User-Agent field, so
   * get into the S3 access logs. Useful for diagnostics.
   * Delegates to {{@link AbstractDelegationTokenBinding#getUserAgentField()}}
   * for the current binding.

View on GitHub (pinned to 2add963021)

Solutions

Put the missing identifier class's jar on the decoding client's classpath
Match hadoop-aws versions between token issuer and consumer
Remove the unusable token from credentials and re-authenticate directly until versions are aligned

Defensive patterns

Strategy: try-catch

Validate before calling

try {
  Class.forName(token.getKind().toString()); // identifier class must be loadable here
} catch (ClassNotFoundException e) {
  credentials.removeToken(token.getService());
  LOG.warn("Removed s3a token whose identifier class {} is not on classpath", token.getKind());
}

Try / catch

try {
  AbstractS3ATokenIdentifier id = delegationTokens.decodeTokenIdentifier(token);
} catch (DelegationTokenIOException e) {
  if (e.getMessage().startsWith("Decoding S3A token")) {
    // missing identifier class: drop token and authenticate directly
    credentials.removeToken(token.getService());
    delegationTokens.bindToAnyDelegationToken();
  } else {
    throw e;
  }
}

Prevention

Match hadoop-aws versions between token issuer and consumers
Ship custom binding identifier jars to all executors
On decode failure, remove the token instead of retrying it

When it happens

Trigger: token.decodeIdentifier() must instantiate the identifier class registered for the token kind; that class is absent from the decoding client's classpath. Common when the token was issued by a different Hadoop version or custom binding whose identifier class the consumer lacks.

Common situations: Cross-version token usage (token issued by newer hadoop-aws, decoded by older); custom binding's identifier jar missing on executors; distro mismatch between token issuer service and compute cluster.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f53483ab88d6253c. Report an issue: GitHub.