{"record":{"id":"31e8510e31d37383","repo":"apache/hadoop","slug":"asyncdataservice-is-already-shutdown","errorCode":null,"errorMessage":"AsyncDataService is already shutdown","messagePattern":"AsyncDataService is already shutdown","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/AsyncDataService.java","lineNumber":67,"sourceCode":"      public Thread newThread(Runnable r) {\n        return new Thread(threadGroup, r);\n      }\n    };\n\n    executor = new ThreadPoolExecutor(CORE_THREADS_PER_VOLUME,\n        MAXIMUM_THREADS_PER_VOLUME, THREADS_KEEP_ALIVE_SECONDS,\n        TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), threadFactory);\n\n    // This can reduce the number of running threads\n    executor.allowCoreThreadTimeOut(true);\n  }\n\n  /**\n   * Execute the task sometime in the future.\n   */\n  synchronized void execute(Runnable task) {\n    if (executor == null) {\n      throw new RuntimeException(\"AsyncDataService is already shutdown\");\n    }\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(\"Current active thread number: \" + executor.getActiveCount()\n          + \" queue size: \" + executor.getQueue().size()\n          + \" scheduled task number: \" + executor.getTaskCount());\n    }\n    executor.execute(task);\n  }\n\n  /**\n   * Gracefully shut down the ThreadPool. Will wait for all data tasks to\n   * finish.\n   */\n  synchronized void shutdown() {\n    if (executor == null) {\n      LOG.warn(\"AsyncDataService has already shut down.\");\n    } else {\n      LOG.info(\"Shutting down all async data service threads...\");","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/AsyncDataService.java#L49-L85","documentation":"AsyncDataService (the NFS gateway's ThreadPoolExecutor wrapper for background write-back tasks) throws an unchecked RuntimeException from execute() if a task is submitted after shutdown() has nulled the executor. It is a lifecycle invariant violation: use-after-shutdown of the async write service.","triggerScenarios":"RpcProgramNfs3 stopDaemons()/shutdown racing with in-flight NFS WRITE requests: a request thread calls execute() after the shutdown thread set executor to null. Also double-stop sequences or tests that close the NFS3 server while client traffic is still flowing.","commonSituations":"NFS gateway shutdown under load; unit/integration tests that start and stop RpcProgramNfs3 without draining requests; custom embedders of Nfs3 that call stop() then keep the RPC dispatcher running.","solutions":["Order shutdown: stop accepting/processing new NFS requests (Nfs3 server) before calling AsyncDataService.shutdown().","If you embed RpcProgramNfs3, synchronize task submission with the service lifecycle and stop submitting before shutdown.","In tests, await termination of the executor and quiesce RPC workers before stop().","As a guard, catch and ignore this RuntimeException only during deliberate shutdown windows, since the task cannot run anyway."],"exampleFix":"// before\nasyncDataService.execute(task); // may throw 'AsyncDataService is already shutdown'\n// after\nif (!asyncDataService.isShutdown()) {\n  asyncDataService.execute(task);\n}","handlingStrategy":"validation","validationCode":"/* inside the same package as AsyncDataService */\nsynchronized void submitSafe(Runnable task) {\n    if (executor != null) {   // guarded by the same lock as execute()/shutdown()\n        executor.execute(task);\n    } else {\n        LOG.debug(\"Dropping task during shutdown: {}\", task);\n    }\n}","typeGuard":"/* lifecycle guard: only submit while the service is alive */\nprivate boolean asyncServiceAlive(AsyncDataService svc) {\n    synchronized (svc) {\n        return svc.executor != null;  // package-private access, same lock as execute()\n    }\n}","tryCatchPattern":"try {\n    asyncDataService.execute(task);\n} catch (RuntimeException e) {\n    if (e.getMessage().contains(\"already shutdown\")) {\n        // we are tearing down: dropping the task is correct, log at debug\n    } else {\n        throw e;\n    }\n}","preventionTips":["Order shutdown: stop the RPC request path (Nfs3) before AsyncDataService.shutdown(), so no new tasks race the nulling of the executor.","Hold the same synchronization discipline as execute() (it is synchronized) when checking/submits.","In tests, awaitTermination on the executor and drain in-flight requests before stopping the service.","Never call execute() from shutdown hooks or metrics flushers that outlive the NFS server."],"tags":["nfs-gateway","hadoop","java","executor","lifecycle","race-condition","shutdown"],"backgroundTag":"executor-after-shutdown","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}