apache/hadoop · critical · IOException

Same delegation token being added twice; invalid entry in fs

Error message

Same delegation token being added twice; invalid entry in fsimage or editlogs

What it means

addPersistedDelegationToken throws IOException when the identifier already exists in currentTokens — the same delegation token appears twice in the persisted state. Because replay is idempotent-by-contract here (fsimage tokens followed by edit-log entries for the same token), a duplicate means the fsimage and edit logs overlap or the logs themselves contain the token twice, i.e., checkpoint/editlog inconsistency. Hadoop treats it as metadata corruption rather than silently merging.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java:299

          "Can't add persisted delegation token to a running SecretManager.");
    }
    int keyId = identifier.getMasterKeyId();
    DelegationKey dKey = allKeys.get(keyId);
    if (dKey == null) {
      LOG
          .warn("No KEY found for persisted identifier "
              + identifier.toString());
      return;
    }
    byte[] password = createPassword(identifier.getBytes(), dKey.getKey());
    if (identifier.getSequenceNumber() > this.delegationTokenSequenceNumber) {
      this.delegationTokenSequenceNumber = identifier.getSequenceNumber();
    }
    if (currentTokens.get(identifier) == null) {
      currentTokens.put(identifier, new DelegationTokenInformation(expiryTime,
          password, getTrackingIdIfEnabled(identifier)));
    } else {
      throw new IOException(
          "Same delegation token being added twice; invalid entry in fsimage or editlogs");
    }
  }

  /**
   * Add a MasterKey to the list of keys.
   * 
   * @param key DelegationKey
   * @throws IOException
   */
  public synchronized void updatePersistedMasterKey(DelegationKey key)
      throws IOException {
    addKey(key);
  }
  
  /**
   * Update the token cache with renewal record in edit logs.
   *

View on GitHub (pinned to 2add963021)

Solutions

Start the NameNode with recovery to reconcile state: 'hdfs namenode -recover' (recover mode) and accept the option to discard inconsistent/duplicate transactions.
Ensure the checkpoint used is at least as new as the earliest retained edit log (name/current fsimage txid <= first edit log txid); re-take a checkpoint if not.
Inspect for duplicated segments across configured dfs.namenode.name.dir entries and keep exactly one consistent set of storage dirs; remove stale duplicated dirs.
After cleanup, 'hdfs oev'-verify the suspect edit logs and re-run NN startup; then create a fresh checkpoint.

Example fix

# before
# IOException: Same delegation token being added twice; invalid entry in fsimage or editlogs

# after: recover, then re-checkpoint so image and edits no longer overlap
hdfs namenode -recover   # choose 'discard conflicting transactions'
hdfs namenode -finalize  # or start NN normally and take a checkpoint
hdfs dfsadmin -saveNamespace

Defensive patterns

Strategy: try-catch

Validate before calling

// Before starting NN: verify no overlap between image txid and first edit log
long imageTxId = FSImageUtil.readCheckpointTxId(currentDir);
long firstEditTxId = FileJournalManager.matchEditLogs(firstLogSegFile);
if (firstEditTxId <= imageTxId) {
  // overlapping state -> duplicate token replay likely; fix storage layout first
}

Try / catch

try {
  nn.start(); // image load + edit replay
} catch (IOException e) {
  if (e.getMessage().contains("Same delegation token being added twice")) {
    // metadata inconsistency: run 'hdfs namenode -recover' to discard duplicates,
    // then re-checkpoint; do NOT force-start past it
  } else { throw e; }
}

Prevention

Always start the NN with a checkpoint at least as new as the earliest edit log.
Keep exactly one consistent set of name.dir storage dirs; delete stale duplicates.
After manual checkpoint imports, verify with 'hdfs oev'/'fsck' before startup.

When it happens

Trigger: Loading a fsimage that already contains a token, then replaying edit logs that re-apply OP_DELEGATION_TOKEN for the same identifier (checkpoint txid behind the edits); or an edit log duplicated/journal misconfiguration causing the same segment to be replayed twice.

Common situations: NameNode started with a checkpoint whose txid is older than the first edit log (image copied manually, 2nn out of sync); journal dirs containing duplicated segments from botched recovery; '-importCheckpoint' picking an image that overlaps available edits; mixing storage dirs from different NN runs.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f74b9f26f67af13a. Report an issue: GitHub.