apache/hadoop · critical · ServerException

S09

S09

Error message

Could not set service [{0}] programmatically -server shutting down-, {1}

What it means

Server.setService(Class, Service) is HttpFS's programmatic hot-swap API: it destroys the currently registered service for an interface and calls init() on the replacement. Error S09 ('Could not set service programmatically -server shutting down-') is thrown when the replacement's init() throws; by then the server has already called destroy() on itself, so the server is down and must be restarted.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java:767

      throw new IllegalStateException("Server shutting down");
    }
    try {
      Service newService = klass.newInstance();
      Service oldService = services.get(newService.getInterface());
      if (oldService != null) {
        try {
          oldService.destroy();
        } catch (Throwable ex) {
          log.error("Could not destroy service [{}], {}",
                    new Object[]{oldService.getInterface(), ex.getMessage(), ex});
        }
      }
      newService.init(this);
      services.put(newService.getInterface(), newService);
    } catch (Exception ex) {
      log.error("Could not set service [{}] programmatically -server shutting down-, {}", klass, ex);
      destroy();
      throw new ServerException(ServerException.ERROR.S09, klass, ex.getMessage(), ex);
    }
  }

}

View on GitHub (pinned to 2add963021)

Solutions

1. Fix the underlying init failure shown in the preceding log.error line and the exception message - setService failed because the new service could not initialize
2. Verify every configuration key the replacement service reads (e.g. httpfs.hadoop.* keys) is present before calling setService
3. Remember the server has destroyed itself after this error - restart it; do not retry setService on the dead instance
4. Test the replacement service's init() against a scratch Server before hot-swapping on a live one

Example fix

// before: replacement service with missing config, init() throws -> server shuts down
server.setService(FileSystemAccess.class, new FileSystemAccessService());

// after: configure the service before injecting it (or use a fully initialized instance)
Configuration conf = server.getConfig();
conf.set("httpfs.hadoop.config.dir", "/etc/hadoop/conf");
conf.set("httpfs.hadoop.authentication.type", "simple");
server.setService(FileSystemAccess.class, new FileSystemAccessService());

// Validate the replacement's init on a throwaway server before hot-swapping
Server scratch = new Server("httpfs", config, null);
try {
  newService.init(scratch);
  newService.destroy();
} catch (Exception ex) {
  throw new IllegalStateException("replacement service not ready", ex);
}

try {
  server.setService(FileSystemAccess.class, newService);
} catch (ServerException ex) {
  if (ex.getError() == ServerException.ERROR.S09) {
  	// server has destroyed itself - restart, do not retry setService
    log.error("setService failed, server shut down: {}", ex.getMessage(), ex);
    restartHttpfs();
  }
}

When it happens

Trigger: Calling server.setService(SomeService.class, newService) where newService.init(this) throws - e.g. a replacement FileSystemAccessService with missing kerberos/hadoop-conf configuration, or a mock service in a test whose init fails; any Exception (other than during old-service destroy, which is only logged) triggers S09 after destroy().

Common situations: Integration tests injecting mock services that are not fully configured; runtime replacement of FileSystemAccessService with one pointing at a wrong hadoop config dir; using setService against a server whose service dependencies were already destroyed.

Related errors

• Service {} is in wrong state: {}
• Bad rule definition: {bad_lines}
• Storage directory is in use.
• Failed to parse "{str}" for the parameter {varName}. The va
• Invalid value: "{str}" does not belong to the domain {domain

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6d8ae6ad0d298606. Report an issue: GitHub.