{"record":{"id":"517292162e2650fe","repo":"apache/hadoop","slug":"got-access-token-error-status-message-message","errorCode":null,"errorMessage":"Got access token error, status message ${message}, ${logInfo}","messagePattern":"Got access token error, status message (.+?), (.+?)","errorType":"exception","errorClass":"InvalidBlockTokenException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtoUtil.java","lineNumber":111,"sourceCode":"      DataTransferTraceInfoProto.Builder traceInfoProtoBuilder =\n          DataTransferTraceInfoProto.newBuilder().setSpanContext(\n              TraceUtils.spanContextToByteString(span.getContext()));\n      builder.setTraceInfo(traceInfoProtoBuilder);\n    }\n    return builder.build();\n  }\n\n  public static void checkBlockOpStatus(\n          BlockOpResponseProto response,\n          String logInfo) throws IOException {\n    checkBlockOpStatus(response, logInfo, false);\n  }\n\n  public static void checkBlockOpStatus(BlockOpResponseProto response,\n      String logInfo, boolean checkBlockPinningErr) throws IOException {\n    if (response.getStatus() != Status.SUCCESS) {\n      if (response.getStatus() == Status.ERROR_ACCESS_TOKEN) {\n        throw new InvalidBlockTokenException(\n          \"Got access token error\"\n          + \", status message \" + response.getMessage()\n          + \", \" + logInfo\n        );\n      } else if (checkBlockPinningErr\n          && response.getStatus() == Status.ERROR_BLOCK_PINNED) {\n        throw new BlockPinningException(\n            \"Got error\"\n            + \", status=\" + response.getStatus().name()\n            + \", status message \" + response.getMessage()\n            + \", \" + logInfo\n          );\n      } else {\n        throw new IOException(\n          \"Got error\"\n          + \", status=\" + response.getStatus().name()\n          + \", status message \" + response.getMessage()\n          + \", \" + logInfo","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtoUtil.java#L93-L129","documentation":"DataTransferProtoUtil.checkBlockOpStatus() inspects the DataNode's BlockOpResponseProto; status ERROR_ACCESS_TOKEN is translated to InvalidBlockTokenException. It means the block access token presented in the op request was rejected: expired, malformed, signed by a different NameNode (post-failover), or security misconfiguration between client, NameNode, and DataNode.","triggerScenarios":"Read/write block ops where the token went stale: long GC pause or idle gaps between getting located blocks with tokens and using them against the DataNode, NameNode failover invalidating keys (block keys roll; DataNode not yet re-registered), or Kerberos/security disabled on one side only.","commonSituations":"HA failover during a job; clients caching LocatedBlocks too long; clock skew; dfs.block.access.key.update.interval/dfs.block.access.token.lifetime too small for slow jobs; mixed secure/insecure cluster nodes.","solutions":["Retry the whole operation: fetch fresh located blocks (which carry new tokens) and re-open the block reader — DFSClient itself does several token retries, so surfacing this usually means exhaustion.","After HA failover, let the client re-resolve the active NameNode and discard cached block locations/tokens.","Check Kerberos/security config is uniform (hadoop.security.authentication) on client, NameNode, and DataNodes, and that block token lifetimes exceed your longest GC/idle window.","Verify DataNodes have heartbeated to the current active NameNode so they hold the current block keys."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  readBlock(...);\n} catch (InvalidBlockTokenException e) {\n  // token stale after failover/pause: drop cached LocatedBlocks, re-fetch, retry once\n  refreshLocatedBlocksAndRetry();\n}","preventionTips":["Do not cache LocatedBlocks across long idle periods; re-fetch before reading.","Keep block token lifetimes (dfs.block.access.token.lifetime) comfortably above worst-case GC/idle pauses.","Run uniform security config on client, NameNode and DataNodes; verify DataNodes are heartbeating to the active NameNode after failover."],"tags":["hdfs","security","block-token","kerberos","ha-failover"],"backgroundTag":"access-token-invalid","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}