apache/hadoop · warning · IOException

Shutdown already in progress.

Error message

Shutdown already in progress.

What it means

shutdownDatanode is a one-shot operation guarded by the shutdownInProgress flag. The first successful call sets the flag and spawns the async shutdown thread (which sleeps ~1s when not upgrading) so the RPC response can be returned. Any second call before the process exits is rejected with this IOException to prevent starting the shutdown thread twice.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:3883

   * @throws IOException throw IOException if not yet initialized.
   */
  private void checkStorageState(String methodName) throws IOException {
    if (data == null) {
      String message = "Storage not yet initialized for " + methodName;
      LOG.debug(message);
      throw new IOException(message);
    }
  }

  @Override // ClientDatanodeProtocol
  public synchronized void shutdownDatanode(boolean forUpgrade) throws IOException {
    checkSuperuserPrivilege();
    LOG.info("shutdownDatanode command received (upgrade={}). " +
        "Shutting down Datanode...", forUpgrade);

    // Shutdown can be called only once.
    if (shutdownInProgress) {
      throw new IOException("Shutdown already in progress.");
    }
    shutdownInProgress = true;
    shutdownForUpgrade = forUpgrade;

    // Asynchronously start the shutdown process so that the rpc response can be
    // sent back.
    Thread shutdownThread = new Thread("Async datanode shutdown thread") {
      @Override public void run() {
        if (!shutdownForUpgrade) {
          // Delay the shutdown a bit if not doing for restart.
          try {
            Thread.sleep(1000);
          } catch (InterruptedException ie) { }
        }
        shutdown();
      }
    };

View on GitHub (pinned to 2add963021)

Solutions

  1. Treat this message as success - the shutdown you wanted is already underway; just wait for the process to exit
  2. Make automation idempotent: check the DN process/RPC state before issuing shutdown again
  3. Accept this specific IOException as a no-op in client code instead of failing the job

Example fix

// before
client.shutdownDatanode(false); // second call throws 'Shutdown already in progress.'

// after
try {
  client.shutdownDatanode(false);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Shutdown already in progress")) {
    LOG.info("Shutdown already requested; nothing to do");
    return;
  }
  throw e;
}
Defensive patterns

Strategy: try-catch

Validate before calling

// Before requesting shutdown, check the DN is still reachable/alive
boolean alive = rpcProxyUnderlyingIsHealthy(); // e.g. JMX heartbeat or rpc ping
if (!alive) { /* already down; skip shutdown call */ }

Try / catch

try {
  client.shutdownDatanode(forUpgrade);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Shutdown already in progress")) {
    return; // idempotent success
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling ClientDatanodeProtocol.shutdownDatanode (or shutdownDatanode forUpgrade) a second time - e.g. two operators/scripts issuing 'dfsadmin -shutdownDatanode', or a client retrying an RPC whose response was lost although the request already took effect.

Common situations: Duplicate ops scripts both shutting down the same node; RPC timeout triggering a client retry after the first shutdown was already accepted; rolling-upgrade tooling racing a manual shutdown.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/47d3c8b1992e5ab9. Report an issue: GitHub.