apache/hadoop · error · InvalidBlockTokenException

Got access token error, status message ${message}, ${logInfo

Error message

Got access token error, status message ${message}, ${logInfo}

What it means

DataTransferProtoUtil.checkBlockOpStatus() inspects the DataNode's BlockOpResponseProto; status ERROR_ACCESS_TOKEN is translated to InvalidBlockTokenException. It means the block access token presented in the op request was rejected: expired, malformed, signed by a different NameNode (post-failover), or security misconfiguration between client, NameNode, and DataNode.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtoUtil.java:111

      DataTransferTraceInfoProto.Builder traceInfoProtoBuilder =
          DataTransferTraceInfoProto.newBuilder().setSpanContext(
              TraceUtils.spanContextToByteString(span.getContext()));
      builder.setTraceInfo(traceInfoProtoBuilder);
    }
    return builder.build();
  }

  public static void checkBlockOpStatus(
          BlockOpResponseProto response,
          String logInfo) throws IOException {
    checkBlockOpStatus(response, logInfo, false);
  }

  public static void checkBlockOpStatus(BlockOpResponseProto response,
      String logInfo, boolean checkBlockPinningErr) throws IOException {
    if (response.getStatus() != Status.SUCCESS) {
      if (response.getStatus() == Status.ERROR_ACCESS_TOKEN) {
        throw new InvalidBlockTokenException(
          "Got access token error"
          + ", status message " + response.getMessage()
          + ", " + logInfo
        );
      } else if (checkBlockPinningErr
          && response.getStatus() == Status.ERROR_BLOCK_PINNED) {
        throw new BlockPinningException(
            "Got error"
            + ", status=" + response.getStatus().name()
            + ", status message " + response.getMessage()
            + ", " + logInfo
          );
      } else {
        throw new IOException(
          "Got error"
          + ", status=" + response.getStatus().name()
          + ", status message " + response.getMessage()
          + ", " + logInfo

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the whole operation: fetch fresh located blocks (which carry new tokens) and re-open the block reader — DFSClient itself does several token retries, so surfacing this usually means exhaustion.
2. After HA failover, let the client re-resolve the active NameNode and discard cached block locations/tokens.
3. Check Kerberos/security config is uniform (hadoop.security.authentication) on client, NameNode, and DataNodes, and that block token lifetimes exceed your longest GC/idle window.
4. Verify DataNodes have heartbeated to the current active NameNode so they hold the current block keys.

try {
  readBlock(...);
} catch (InvalidBlockTokenException e) {
  // token stale after failover/pause: drop cached LocatedBlocks, re-fetch, retry once
  refreshLocatedBlocksAndRetry();
}

When it happens

Trigger: Read/write block ops where the token went stale: long GC pause or idle gaps between getting located blocks with tokens and using them against the DataNode, NameNode failover invalidating keys (block keys roll; DataNode not yet re-registered), or Kerberos/security disabled on one side only.

Common situations: HA failover during a job; clients caching LocatedBlocks too long; clock skew; dfs.block.access.key.update.interval/dfs.block.access.token.lifetime too small for slow jobs; mixed secure/insecure cluster nodes.

Related errors

• Fetch of delegation token failed
• Block pool {bpid} is not found
• currentKey hasn't been initialized.
• Can't re-compute encryption key for nonce, since the require
• Cannot get access token since BlockKeyUpdater is not running

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/517292162e2650fe. Report an issue: GitHub.