{"record":{"id":"47d3c8b1992e5ab9","repo":"apache/hadoop","slug":"shutdown-already-in-progress","errorCode":null,"errorMessage":"Shutdown already in progress.","messagePattern":"Shutdown already in progress\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java","lineNumber":3883,"sourceCode":"   * @throws IOException throw IOException if not yet initialized.\n   */\n  private void checkStorageState(String methodName) throws IOException {\n    if (data == null) {\n      String message = \"Storage not yet initialized for \" + methodName;\n      LOG.debug(message);\n      throw new IOException(message);\n    }\n  }\n\n  @Override // ClientDatanodeProtocol\n  public synchronized void shutdownDatanode(boolean forUpgrade) throws IOException {\n    checkSuperuserPrivilege();\n    LOG.info(\"shutdownDatanode command received (upgrade={}). \" +\n        \"Shutting down Datanode...\", forUpgrade);\n\n    // Shutdown can be called only once.\n    if (shutdownInProgress) {\n      throw new IOException(\"Shutdown already in progress.\");\n    }\n    shutdownInProgress = true;\n    shutdownForUpgrade = forUpgrade;\n\n    // Asynchronously start the shutdown process so that the rpc response can be\n    // sent back.\n    Thread shutdownThread = new Thread(\"Async datanode shutdown thread\") {\n      @Override public void run() {\n        if (!shutdownForUpgrade) {\n          // Delay the shutdown a bit if not doing for restart.\n          try {\n            Thread.sleep(1000);\n          } catch (InterruptedException ie) { }\n        }\n        shutdown();\n      }\n    };\n","sourceCodeStart":3865,"sourceCodeEnd":3901,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L3865-L3901","documentation":"shutdownDatanode is a one-shot operation guarded by the shutdownInProgress flag. The first successful call sets the flag and spawns the async shutdown thread (which sleeps ~1s when not upgrading) so the RPC response can be returned. Any second call before the process exits is rejected with this IOException to prevent starting the shutdown thread twice.","triggerScenarios":"Calling ClientDatanodeProtocol.shutdownDatanode (or shutdownDatanode forUpgrade) a second time - e.g. two operators/scripts issuing 'dfsadmin -shutdownDatanode', or a client retrying an RPC whose response was lost although the request already took effect.","commonSituations":"Duplicate ops scripts both shutting down the same node; RPC timeout triggering a client retry after the first shutdown was already accepted; rolling-upgrade tooling racing a manual shutdown.","solutions":["Treat this message as success - the shutdown you wanted is already underway; just wait for the process to exit","Make automation idempotent: check the DN process/RPC state before issuing shutdown again","Accept this specific IOException as a no-op in client code instead of failing the job"],"exampleFix":"// before\nclient.shutdownDatanode(false); // second call throws 'Shutdown already in progress.'\n\n// after\ntry {\n  client.shutdownDatanode(false);\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"Shutdown already in progress\")) {\n    LOG.info(\"Shutdown already requested; nothing to do\");\n    return;\n  }\n  throw e;\n}","handlingStrategy":"try-catch","validationCode":"// Before requesting shutdown, check the DN is still reachable/alive\nboolean alive = rpcProxyUnderlyingIsHealthy(); // e.g. JMX heartbeat or rpc ping\nif (!alive) { /* already down; skip shutdown call */ }","typeGuard":null,"tryCatchPattern":"try {\n  client.shutdownDatanode(forUpgrade);\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"Shutdown already in progress\")) {\n    return; // idempotent success\n  }\n  throw e;\n}","preventionTips":["Make shutdown scripts idempotent - one request per node, plus tolerate the 'already in progress' reply","Do not blind-retry shutdown RPCs after timeouts; the first request usually took effect","Coordinate operators/tooling so only one shutdown trigger runs per datanode"],"tags":["hadoop","hdfs","datanode","shutdown","idempotency","rpc"],"backgroundTag":"duplicate-shutdown-request","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}