apache/hadoop · error · IllegalStateException

Cannot load data from ZooKeeper; itwas written with a newer

Error message

Cannot load data from ZooKeeper; itwas written with a newer version

What it means

ZKSignerSecretProvider serializes secrets into a znode with a leading data-format version integer. On pullFromZK(), if the stored version is greater than the DATA_VERSION this JVM's hadoop-auth was built with, it throws IllegalStateException('Cannot load data from ZooKeeper; itwas written with a newer version') — a forward-compatibility guard, since older code cannot safely interpret newer payloads.

Source

Thrown at hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:313

    bb.putLong(nextRolloverDate);
    return bb.array();
  }

  /**
   * Pulls data from ZooKeeper.  If isInit is false, it will only parse the
   * next secret and version.  If isInit is true, it will also parse the current
   * and previous secrets, and the next rollover date; it will also init the
   * secrets.  Hence, isInit should only be true on startup.
   * @param isInit  see description above
   */
  private synchronized void pullFromZK(boolean isInit) {
    try {
      Stat stat = new Stat();
      byte[] bytes = client.getData().storingStatIn(stat).forPath(path);
      ByteBuffer bb = ByteBuffer.wrap(bytes);
      int dataVersion = bb.getInt();
      if (dataVersion > DATA_VERSION) {
        throw new IllegalStateException("Cannot load data from ZooKeeper; it"
                + "was written with a newer version");
      }
      int nextSecretLength = bb.getInt();
      byte[] nextSecret = new byte[nextSecretLength];
      bb.get(nextSecret);
      this.nextSecret = nextSecret;
      zkVersion = stat.getVersion();
      if (isInit) {
        int currentSecretLength = bb.getInt();
        byte[] currentSecret = new byte[currentSecretLength];
        bb.get(currentSecret);
        int previousSecretLength = bb.getInt();
        byte[] previousSecret = null;
        if (previousSecretLength > 0) {
          previousSecret = new byte[previousSecretLength];
          bb.get(previousSecret);
        }
        super.initSecrets(currentSecret, previousSecret);

View on GitHub (pinned to 2add963021)

Solutions

Complete the rolling upgrade so every node reading that znode runs the newer hadoop-auth version
Or, during upgrade windows, point nodes at a fresh znode path (signer.secret.provider.zookeeper.path) to start with current-format data, accepting re-authentication
Do not permanently run mixed versions against one secret znode

Example fix

# before: old node reads znode written by new node
signer.secret.provider.zookeeper.path=/hadoop-auth-secret

# after (during upgrade): isolate old fleet on a fresh path
signer.secret.provider.zookeeper.path=/hadoop-auth-secret-v2

Defensive patterns

Strategy: fallback

Validate before calling

// cannot pre-validate remotely stored bytes; after getData, provider checks version itself — mirror the check when reading custom ZK data
if (dataVersion > DATA_VERSION) throw new IllegalStateException("newer znode data");

Try / catch

try { /* provider init reads znode */ } catch (IllegalStateException e) { /* mixed-version cluster: finish upgrade or switch to a fresh znode path */ }

Prevention

Avoid permanently mixed Hadoop versions sharing one secret znode
During rolling upgrades, plan a fresh znode path or complete the upgrade before re-enabling nodes
Keep dev/test clusters pointed at their own ZooKeeper namespace

When it happens

Trigger: A cluster node running an older Hadoop version connects to a ZooKeeper znode whose secret data was written by an upgraded (newer) node; mixed-version rolling upgrade while the zookeeper secret provider is active.

Common situations: Rolling upgrades where some RMS/UI nodes are upgraded first and write the new-format znode; a test/dev environment pointing at a shared ZooKeeper used by newer software.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2f65fd0739f3fa0e. Report an issue: GitHub.