{"record":{"id":"f74b9f26f67af13a","repo":"apache/hadoop","slug":"same-delegation-token-being-added-twice-invalid-e","errorCode":null,"errorMessage":"Same delegation token being added twice; invalid entry in fsimage or editlogs","messagePattern":"Same delegation token being added twice; invalid entry in fsimage or editlogs","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java","lineNumber":299,"sourceCode":"          \"Can't add persisted delegation token to a running SecretManager.\");\n    }\n    int keyId = identifier.getMasterKeyId();\n    DelegationKey dKey = allKeys.get(keyId);\n    if (dKey == null) {\n      LOG\n          .warn(\"No KEY found for persisted identifier \"\n              + identifier.toString());\n      return;\n    }\n    byte[] password = createPassword(identifier.getBytes(), dKey.getKey());\n    if (identifier.getSequenceNumber() > this.delegationTokenSequenceNumber) {\n      this.delegationTokenSequenceNumber = identifier.getSequenceNumber();\n    }\n    if (currentTokens.get(identifier) == null) {\n      currentTokens.put(identifier, new DelegationTokenInformation(expiryTime,\n          password, getTrackingIdIfEnabled(identifier)));\n    } else {\n      throw new IOException(\n          \"Same delegation token being added twice; invalid entry in fsimage or editlogs\");\n    }\n  }\n\n  /**\n   * Add a MasterKey to the list of keys.\n   * \n   * @param key DelegationKey\n   * @throws IOException\n   */\n  public synchronized void updatePersistedMasterKey(DelegationKey key)\n      throws IOException {\n    addKey(key);\n  }\n  \n  /**\n   * Update the token cache with renewal record in edit logs.\n   * ","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java#L281-L317","documentation":"addPersistedDelegationToken throws IOException when the identifier already exists in currentTokens — the same delegation token appears twice in the persisted state. Because replay is idempotent-by-contract here (fsimage tokens followed by edit-log entries for the same token), a duplicate means the fsimage and edit logs overlap or the logs themselves contain the token twice, i.e., checkpoint/editlog inconsistency. Hadoop treats it as metadata corruption rather than silently merging.","triggerScenarios":"Loading a fsimage that already contains a token, then replaying edit logs that re-apply OP_DELEGATION_TOKEN for the same identifier (checkpoint txid behind the edits); or an edit log duplicated/journal misconfiguration causing the same segment to be replayed twice.","commonSituations":"NameNode started with a checkpoint whose txid is older than the first edit log (image copied manually, 2nn out of sync); journal dirs containing duplicated segments from botched recovery; '-importCheckpoint' picking an image that overlaps available edits; mixing storage dirs from different NN runs.","solutions":["Start the NameNode with recovery to reconcile state: 'hdfs namenode -recover' (recover mode) and accept the option to discard inconsistent/duplicate transactions.","Ensure the checkpoint used is at least as new as the earliest retained edit log (name/current fsimage txid <= first edit log txid); re-take a checkpoint if not.","Inspect for duplicated segments across configured dfs.namenode.name.dir entries and keep exactly one consistent set of storage dirs; remove stale duplicated dirs.","After cleanup, 'hdfs oev'-verify the suspect edit logs and re-run NN startup; then create a fresh checkpoint."],"exampleFix":"# before\n# IOException: Same delegation token being added twice; invalid entry in fsimage or editlogs\n\n# after: recover, then re-checkpoint so image and edits no longer overlap\nhdfs namenode -recover   # choose 'discard conflicting transactions'\nhdfs namenode -finalize  # or start NN normally and take a checkpoint\nhdfs dfsadmin -saveNamespace","handlingStrategy":"try-catch","validationCode":"// Before starting NN: verify no overlap between image txid and first edit log\nlong imageTxId = FSImageUtil.readCheckpointTxId(currentDir);\nlong firstEditTxId = FileJournalManager.matchEditLogs(firstLogSegFile);\nif (firstEditTxId <= imageTxId) {\n  // overlapping state -> duplicate token replay likely; fix storage layout first\n}","typeGuard":null,"tryCatchPattern":"try {\n  nn.start(); // image load + edit replay\n} catch (IOException e) {\n  if (e.getMessage().contains(\"Same delegation token being added twice\")) {\n    // metadata inconsistency: run 'hdfs namenode -recover' to discard duplicates,\n    // then re-checkpoint; do NOT force-start past it\n  } else { throw e; }\n}","preventionTips":["Always start the NN with a checkpoint at least as new as the earliest edit log.","Keep exactly one consistent set of name.dir storage dirs; delete stale duplicates.","After manual checkpoint imports, verify with 'hdfs oev'/'fsck' before startup."],"tags":["hdfs","delegation-token","fsimage","edit-log","corruption","startup"],"backgroundTag":"fsimage-editlog-inconsistency","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}