apache/hadoop · critical · ServiceFailedException

Couldn't transition to active

Error message

Couldn't transition to active

What it means

ZKFailoverController.becomeActive() wraps any non-ServiceFailedException Throwable thrown while making the local service active into ServiceFailedException("Couldn't transition to active", t). The local service (typically a NameNode) refused or failed the transitionToActive call and the ZKFC reports the failure after recording a failed ActiveAttemptRecord.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java:433

          conf, FailoverController.getRpcTimeoutToNewActive(conf)),
          createReqInfo());
      String msg = "Successfully transitioned " + localTarget +
          " to active state";
      LOG.info(msg);
      serviceState = HAServiceState.ACTIVE;
      recordActiveAttempt(new ActiveAttemptRecord(true, msg));

    } catch (Throwable t) {
      String msg = "Couldn't make " + localTarget + " active";
      LOG.error(msg, t);
      
      recordActiveAttempt(new ActiveAttemptRecord(false, msg + "\n" +
          StringUtils.stringifyException(t)));

      if (t instanceof ServiceFailedException) {
        throw (ServiceFailedException)t;
      } else {
        throw new ServiceFailedException("Couldn't transition to active",
            t);
      }
/*
* TODO:
* we need to make sure that if we get fenced and then quickly restarted,
* none of these calls will retry across the restart boundary
* perhaps the solution is that, whenever the nn starts, it gets a unique
* ID, and when we start becoming active, we record it, and then any future
* calls use the same ID
*/
      
    }
  }

  /**
   * Store the results of the last attempt to become active.
   * This is used so that, during manually initiated failover,
   * we can report back the results of the attempt to become active

View on GitHub (pinned to 2add963021)

Solutions

1. Read the wrapped cause: the preceding log 'Couldn't make <target> active' plus the NameNode's own log states the real reason.
2. Ensure all JournalNodes are up and the local NameNode can write to the shared edits (check JN process, ports, dfs.journalnode.* addresses).
3. If the NN is not ready yet, let the ZKFC retry — election will be re-attempted; fix the NN-reported blocker first.
4. Verify local storage dirs are writable and not full.

// Before relying on election: verify the local NN can write shared edits
// e.g. ensure JournalNodes are up from the NN host:
//   for jn in jn1 jn2 jn3: nc -z $jn 8485
// And check local storage dirs writable. There is no public API that
// 'pre-flights' transitionToActive without side effects.

try {
  zkfcInitiatedElection(); // or rely on ZKFC retry loop
} catch (ServiceFailedException sfe) {
  if ("Couldn't transition to active".equals(sfe.getMessage())) {
    Throwable root = sfe.getCause(); // the NN's real failure
    // fix JournalNodes/storage per root cause; ZKFC retries election
  }
}

When it happens

Trigger: The local service's transitionToActive throws an IOException or RuntimeException rather than ServiceFailedException — e.g. JournalNode quorum unavailable so the NN cannot start writing edits, NN not ready / still starting, storage or shared-edits misconfiguration on the local node.

Common situations: JournalNodes down when this ZKFC wins the election; dfs.namenode.shared.edits.dir wrong on the local node; NN transitioning slowly (large image/edits) and internal calls time out; disk full on the edits dir.

Related errors

• Unable to failover to {}
• hadoop.security.authorizationis configured to true but servi
• Bad argument: {}
• ZK Failover Controller failed: {}
• Unable to fence {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/faa6543fc797cc0d. Report an issue: GitHub.