apache/hadoop · error · RuntimeException

AsyncDataService is already shutdown

Error message

AsyncDataService is already shutdown

What it means

AsyncDataService (the NFS gateway's ThreadPoolExecutor wrapper for background write-back tasks) throws an unchecked RuntimeException from execute() if a task is submitted after shutdown() has nulled the executor. It is a lifecycle invariant violation: use-after-shutdown of the async write service.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/AsyncDataService.java:67

      public Thread newThread(Runnable r) {
        return new Thread(threadGroup, r);
      }
    };

    executor = new ThreadPoolExecutor(CORE_THREADS_PER_VOLUME,
        MAXIMUM_THREADS_PER_VOLUME, THREADS_KEEP_ALIVE_SECONDS,
        TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), threadFactory);

    // This can reduce the number of running threads
    executor.allowCoreThreadTimeOut(true);
  }

  /**
   * Execute the task sometime in the future.
   */
  synchronized void execute(Runnable task) {
    if (executor == null) {
      throw new RuntimeException("AsyncDataService is already shutdown");
    }
    if (LOG.isDebugEnabled()) {
      LOG.debug("Current active thread number: " + executor.getActiveCount()
          + " queue size: " + executor.getQueue().size()
          + " scheduled task number: " + executor.getTaskCount());
    }
    executor.execute(task);
  }

  /**
   * Gracefully shut down the ThreadPool. Will wait for all data tasks to
   * finish.
   */
  synchronized void shutdown() {
    if (executor == null) {
      LOG.warn("AsyncDataService has already shut down.");
    } else {
      LOG.info("Shutting down all async data service threads...");

View on GitHub (pinned to 2add963021)

Solutions

1. Order shutdown: stop accepting/processing new NFS requests (Nfs3 server) before calling AsyncDataService.shutdown().
2. If you embed RpcProgramNfs3, synchronize task submission with the service lifecycle and stop submitting before shutdown.
3. In tests, await termination of the executor and quiesce RPC workers before stop().
4. As a guard, catch and ignore this RuntimeException only during deliberate shutdown windows, since the task cannot run anyway.

Example fix

// before
asyncDataService.execute(task); // may throw 'AsyncDataService is already shutdown'
// after
if (!asyncDataService.isShutdown()) {
  asyncDataService.execute(task);
}

/* inside the same package as AsyncDataService */
synchronized void submitSafe(Runnable task) {
    if (executor != null) {   // guarded by the same lock as execute()/shutdown()
        executor.execute(task);
    } else {
        LOG.debug("Dropping task during shutdown: {}", task);
    }
}

/* lifecycle guard: only submit while the service is alive */
private boolean asyncServiceAlive(AsyncDataService svc) {
    synchronized (svc) {
        return svc.executor != null;  // package-private access, same lock as execute()
    }
}

try {
    asyncDataService.execute(task);
} catch (RuntimeException e) {
    if (e.getMessage().contains("already shutdown")) {
        // we are tearing down: dropping the task is correct, log at debug
    } else {
        throw e;
    }
}

When it happens

Trigger: RpcProgramNfs3 stopDaemons()/shutdown racing with in-flight NFS WRITE requests: a request thread calls execute() after the shutdown thread set executor to null. Also double-stop sequences or tests that close the NFS3 server while client traffic is still flowing.

Common situations: NFS gateway shutdown under load; unit/integration tests that start and stop RpcProgramNfs3 without draining requests; custom embedders of Nfs3 that call stop() then keep the RPC dispatcher running.

Related errors

• Shutdown in progress, cannot add a shutdownHook
• Shutdown in progress, cannot remove a shutdownHook
• key + ": Stream is closed!"
• Stream closed
• Multipart upload incomplete: expected {} parts but got {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/31e8510e31d37383. Report an issue: GitHub.