apache/hadoop · error · IOException

Wait Interrupted

Error message

Wait Interrupted

What it means

DFSAdmin.waitExitSafeMode(DistributedFileSystem, boolean) implements `hdfs dfsadmin -safemode wait`: it sleeps 5 seconds and re-queries setSafeMode(SAFEMODE_GET) until the NameNode leaves safe mode. If the calling thread is interrupted during Thread.sleep, the InterruptedException is swallowed and rethrown as IOException('Wait Interrupted'); the thread's interrupted flag is NOT restored, so interruption state is lost.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java:738

      }
    } else {
      boolean inSafeMode = dfs.setSafeMode(action);
      if (waitExitSafe) {
        inSafeMode = waitExitSafeMode(dfs, inSafeMode);
      }
      System.out.println("Safe mode is " + (inSafeMode ? "ON" : "OFF"));
    }

  }

  @SuppressWarnings("deprecation")
  private boolean waitExitSafeMode(DistributedFileSystem dfs, boolean inSafeMode)
      throws IOException {
    while (inSafeMode) {
      try {
        Thread.sleep(5000);
      } catch (java.lang.InterruptedException e) {
        throw new IOException("Wait Interrupted");
      }
      inSafeMode = dfs.setSafeMode(SafeModeAction.SAFEMODE_GET, false);
    }
    return inSafeMode;
  }

  private boolean waitExitSafeMode(ClientProtocol nn, boolean inSafeMode)
      throws IOException {
    while (inSafeMode) {
      try {
        Thread.sleep(5000);
      } catch (java.lang.InterruptedException e) {
        throw new IOException("Wait Interrupted");
      }
      inSafeMode = nn.setSafeMode(SafeModeAction.SAFEMODE_GET, false);
    }
    return inSafeMode;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Diagnose why safe mode persists first: hdfs dfsadmin -safemode get plus the missing-blocks report (hdfs fsck /), fix blocks or consciously adjust dfs.namenode.safemode.threshold-pct
2. Re-run -safemode wait once the NN is healthy
3. If embedding DFSAdmin, catch IOException and restore the flag: Thread.currentThread().interrupt() when the message is 'Wait Interrupted'
4. Prefer your own bounded poll of setSafeMode(SAFEMODE_GET) with an explicit timeout instead of the blocking CLI wait

Example fix

// before: blocks forever, loses interruption on shutdown
admin.waitExitSafeMode(dfs, true);

// after: bounded poll that preserves interrupt status
while (dfs.setSafeMode(SafeModeAction.SAFEMODE_GET, false) && deadline > now()) {
  Thread.sleep(5000);
} // catch InterruptedException: Thread.currentThread().interrupt(); break;

null

try {
  dfsadmin.run(new String[]{"-safemode", "wait"});
} catch (IOException e) {
  if ("Wait Interrupted".equals(e.getMessage())) {
    Thread.currentThread().interrupt(); // restore the lost interrupt flag
    throw new ShutdownSignalException(e);
  }
  throw e;
}

When it happens

Trigger: Interrupting dfsadmin -safemode wait while the NN is still in safe mode: container SIGTERM with a grace period, orchestrator timeout killing the CLI, embedding DFSAdmin in an application whose executor shuts down, or Ctrl-C under wrappers that interrupt threads.

Common situations: Safe mode never exits because of missing/under-replicated blocks, and a supervisor timeout kills the wait; systemd stopping the unit; test harnesses cancelling the admin thread; long safe-mode waits after mass datanode loss.

Related errors

• "{}" is not a valid value for a quota.
• Storage type {} is not available. Available storage types ar
• Failed to convert "{}" to RollingUpgradeAction
• "Invalid argument: " + cmd
• Illegal option {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dd8ba2de1e2c4450. Report an issue: GitHub.