apache/hadoop · error · DelegationTokenIOException

Filesystem not generating Delegation Tokens: {url}

Error message

Filesystem not generating Delegation Tokens: {url}

What it means

S3ADtFetcher implements the 's3a' TokenFetcher used by tools like `hadoop fetchdt`: it opens the filesystem for the URL and calls getDelegationToken(renewer). S3AFileSystem returns null when delegation tokens are not enabled (fs.s3a.delegation.token.binding defaults to empty), so the fetcher fails with DelegationTokenIOException FETCH_FAILED 'Filesystem not generating Delegation Tokens: <url>'.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADtFetcher.java:75

  /**
   *  Returns Token object via FileSystem, null if bad argument.
   *  @param conf - a Configuration object used with FileSystem.get()
   *  @param creds - a Credentials object to which token(s) will be added
   *  @param renewer  - the renewer to send with the token request
   *  @param url  - the URL to which the request is sent
   *  @return a Token, or null if fetch fails.
   */
  public Token<?> addDelegationTokens(Configuration conf,
      Credentials creds,
      String renewer,
      String url) throws Exception {
    if (!url.startsWith(getServiceName().toString())) {
      url = getServiceName().toString() + "://" + url;
    }
    FileSystem fs = FileSystem.get(URI.create(url), conf);
    Token<?> token = fs.getDelegationToken(renewer);
    if (token == null) {
      throw new DelegationTokenIOException(FETCH_FAILED + ": " + url);
    }
    creds.addToken(token.getService(), token);
    return token;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Configure a delegation-token binding on the target filesystem: fs.s3a.delegation.token.binding=org.apache.hadoop.fs.s3a.auth.delegation.S3ATokenBinding (full credentials) or org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding (session credentials)
2. Make sure the fetchdt process uses the same configuration as the filesystem instance it contacts (same core-site, same bucket-level overrides)
3. Re-run the fetchdt command after the config change; the FS must be re-instantiated to pick it up

Example fix

<!-- before: DT support off (default) -->
<!-- fs.s3a.delegation.token.binding unset -->

<!-- after -->
<property>
  <name>fs.s3a.delegation.token.binding</name>
  <value>org.apache.hadoop.fs.s3a.auth.delegation.S3ATokenBinding</value>
</property>

String binding = conf.getTrimmed("fs.s3a.delegation.token.binding", "");
if (binding.isEmpty()) {
  throw new IllegalStateException(
      "Cannot fetchdt s3a tokens: set fs.s3a.delegation.token.binding on the target filesystem");
}

try {
  Token<?> t = new S3ADtFetcher().addDelegationTokens(conf, creds, renewer, "s3a://bucket");
} catch (DelegationTokenIOException e) {
  if (e.getMessage().startsWith("fetch failed")) { // FETCH_FAILED
    // DT not enabled on target FS: enable binding or skip token collection
    LOG.error("Enable fs.s3a.delegation.token.binding on the S3A filesystem, then retry");
  }
  throw e;
}

When it happens

Trigger: Running a delegation-token fetch (hadoop fetchdt s3a://bucket, or a KeyingTool/TokenFetcher integration) against an S3A filesystem where fs.s3a.delegation.token.binding is unset, so fs.getDelegationToken(renewer) returns null.

Common situations: Cluster-wide core-site.xml has no delegation binding configured; the fetchdt client's config differs from the filesystem's; trying to collect s3a tokens for a distcp/Hive workflow before enabling DT support.

Related errors

• No AWS credential providers created by Delegation Token Bind
• {component}: Invalid AWS credentials in {credentials} requir
• Class {className} {e} (configuration key fs.s3a.http.signer.
• Invalid format (Expected name, name:SignerClass, name:Signer
• Delegation token is wrong class; expected a token identifier

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9a6d988797fa7d29. Report an issue: GitHub.