apache/hadoop · critical · ServerException

S11

S11

Error message

Service [{0}] exception during status change to [{1}] -server shutting down-, {2}

What it means

Server.setStatus() advances the lifecycle (e.g. to STARTED) and notifies every registered Service via serverStatusChange(); if any service throws, the server logs the failure, calls destroy() (shutting itself down), and rethrows ServerException S11 with the service interface name, target status, and the underlying cause message. S11 is a wrapper — the real fault is the {2} message and the logged stack trace.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java:279

   * @param status status to set.
   *
   * @throws ServerException thrown if the service has been destroy because of
   * a failed notification to a service.
   */
  public void setStatus(Status status) throws ServerException {
    Check.notNull(status, "status");
    if (status.settable) {
      if (status != this.status) {
        Status oldStatus = this.status;
        this.status = status;
        for (Service service : services.values()) {
          try {
            service.serverStatusChange(oldStatus, status);
          } catch (Exception ex) {
            log.error("Service [{}] exception during status change to [{}] -server shutting down-,  {}",
                      new Object[]{service.getInterface().getSimpleName(), status, ex.getMessage(), ex});
            destroy();
            throw new ServerException(ServerException.ERROR.S11, service.getInterface().getSimpleName(),
                                      status, ex.getMessage(), ex);
          }
        }
      }
    } else {
      throw new IllegalArgumentException("Status [" + status + " is not settable");
    }
  }

  /**
   * Verifies the server is operational.
   *
   * @throws IllegalStateException thrown if the server is not operational.
   */
  protected void ensureOperational() {
    if (!getStatus().isOperational()) {
      throw new IllegalStateException("Server is not running");
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Read the full log line above/with the exception: 'Service [<iface>] exception during status change to [<status>]' plus the cause {2} and stack identify the failing service — fix that root cause (e.g. NameNode URI, keytab path, proxy settings).
2. If it is a startup-order issue, add dependency ordering/start retries so HDFS is reachable when HttpFS starts.
3. For custom services, make serverStatusChange() defensive (catch and handle internally, or fail fast with a clear message), since one throwhere takes the whole server down.
4. If the service is optional, remove/disable it from httpfs.services until fixed.

Example fix

// before: custom service that can throw during startup
public void serverStatusChange(Status old, Status status) {
  connectToRemote(); // throws -> whole server shuts down
}

// after
public void serverStatusChange(Status old, Status status) {
  try {
    connectToRemote();
  } catch (Exception e) {
    LOG.warn("remote connect deferred: {}", e.getMessage()); // retry later instead of killing server
  }
}

// embedder: validate service dependencies before setStatus(STARTED)
for (Service s : services) {
  if (!s.getDependencies().stream().allMatch(services::containsKey)) {
    throw new IllegalStateException("missing dependency for " + s.getInterface());
  }
}

try {
  server.setStatus(Server.Status.STARTED);
} catch (ServerException e) {
  if (e.getError() == ServerException.ERROR.S11) {
    // server has already destroyed itself; inspect cause {2} and the 'Service [x] exception' log line
    LOG.error("service failed during status change: {}", e.getMessage(), e.getCause());
    // do not retry blind; fix the named service first
  }
  throw e;
}

When it happens

Trigger: The FileSystemAccess service failing to initialize against HDFS (bad fs.defaultFS, NameNode unreachable at startup, proxy-user misconfiguration); a scheduler/instrumentation service failing during STARTED transition; any custom service registered in httpfs.services throwing from serverStatusChange(); shutdown-phase errors while transitioning to STOPPED.

Common situations: HttpFS starting before the NameNode is available; Kerberos/keytab problems inside a service; custom services added to httpfs.services without handling status transitions; the same root cause appearing once at startup and again at shutdown.

Related errors

• Undefined property: signature.secret.file
• No secret in HttpFs signature secret file: {0}
• Could not read HttpFS signature secret file: {0}
• Call interrupted
• Could not set handler for signal "${name}".This can happen i

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/532c9c9b3d5af2e7. Report an issue: GitHub.