apache/hadoop · critical · IOException

Datanode CTime (={}) is not equal to namenode CTime (={})

Error message

Datanode CTime (={}) is not equal to namenode CTime (={})

What it means

After doTransition reported that no layout transition was needed, the DN found its persisted block pool cTime differs from the NameNode's cTime. DN storage state and namespace state disagree, which in practice means an upgrade/rollback of the namespace happened without the matching DN-side transition.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:185

      case NOT_FORMATTED: // format
        LOG.info("Block pool storage directory for location {} and block pool"
                + " id {} is not formatted. Formatting ...", location,
            nsInfo.getBlockPoolID());
        format(sd, nsInfo);
        break;
      default:  // recovery part is common
        sd.doRecover(curState);
      }

      // 2. Do transitions
      // Each storage directory is treated individually.
      // During startup some of them can upgrade or roll back
      // while others could be up-to-date for the regular startup.
      if (!doTransition(sd, nsInfo, startOpt, callables, conf)) {

        // 3. Check CTime and update successfully loaded storage.
        if (getCTime() != nsInfo.getCTime()) {
          throw new IOException("Datanode CTime (=" + getCTime()
              + ") is not equal to namenode CTime (=" + nsInfo.getCTime() + ")");
        }
        setServiceLayoutVersion(getServiceLayoutVersion());
        writeProperties(sd);
      }

      return sd;
    } catch (IOException ioe) {
      sd.unlock();
      throw ioe;
    }
  }

  /**
   * Analyze and load storage directories. Recover from previous transitions if
   * required.
   *
   * The block pool storages are either all analyzed or none of them is loaded.

View on GitHub (pinned to 2add963021)

Solutions

1. Restart the DN with -upgrade (or -rollback, matching what the namespace went through)
2. If the NN was deliberately re-formatted, delete the DN's stale block pool dirs so they re-register fresh
3. Restore the NN namespace from backup if the cTime change was accidental

Example fix

# before
hdfs --daemon start datanode            # -regular, throws

# after (namespace was upgraded)
hdfs --daemon start datanode -upgrade
# or, if NN rolled back:
hdfs --daemon start datanode -rollback

catch (IOException e) {
  if (e.getMessage().contains("is not equal to namenode CTime")) {
    // restart DN with -upgrade or -rollback matching the namespace's transition
    failFastWithGuidance(e);
  }
}

When it happens

Trigger: NN was upgraded or its cTime otherwise bumped while the DN starts with plain -regular; NN re-formatted; an aborted upgrade left BP storage at the old cTime with no transition path selected.

Common situations: Operators re-format the NN and restart DNs normally; rolling upgrade finalized inconsistently; test clusters reused after NN reset.

Related errors

• Datanode state: LV = {} CTime = {} is newer than the namespa
• Incompatible node types: storageType={storageType} but Stora
• Storage directory for location {} and block pool id {} does
• BlockPoolSliceStorage.recoverTransitionRead: attempt to load
• file VERSION is invalid.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4edbe40ceafb5594. Report an issue: GitHub.