apache/hadoop · warning · IOException

Another reconfiguration task is running.

Error message

Another reconfiguration task is running.

What it means

Thrown by ReconfigurableBase.startReconfigurationTask() when a background ReconfigurationThread is already active for this service (reconfigThread != null under reconfigLock). The class permits only one in-flight reconfiguration at a time, so a second start attempt is rejected with an IOException before any thread is created. The running status is observable via getReconfigurationTaskStatus(), which reports endTime 0 while a task is running.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ReconfigurableBase.java:179

      }
    }
  }

  /**
   * Start a reconfiguration task to reload configuration in background.
   * @throws IOException raised on errors performing I/O.
   */
  public void startReconfigurationTask() throws IOException {
    synchronized (reconfigLock) {
      if (!shouldRun) {
        String errorMessage = "The server is stopped.";
        LOG.warn(errorMessage);
        throw new IOException(errorMessage);
      }
      if (reconfigThread != null) {
        String errorMessage = "Another reconfiguration task is running.";
        LOG.warn(errorMessage);
        throw new IOException(errorMessage);
      }
      reconfigThread = new ReconfigurationThread(this);
      reconfigThread.setDaemon(true);
      reconfigThread.setName("Reconfiguration Task");
      reconfigThread.start();
      startTime = Time.now();
    }
  }

  public ReconfigurationTaskStatus getReconfigurationTaskStatus() {
    synchronized (reconfigLock) {
      if (reconfigThread != null) {
        return new ReconfigurationTaskStatus(startTime, 0, null);
      }
      return new ReconfigurationTaskStatus(startTime, endTime, status);
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Before starting, call getReconfigurationTaskStatus() and only invoke startReconfigurationTask() when no task is running (status startTime==0, or endTime>0 from a completed task)
2. Wait for the in-flight task to finish by polling getReconfigurationTaskStatus() until endTime becomes non-zero, then start the new task
3. If the prior task is stuck, call shutdownReconfigurationTask() (it joins the thread and nulls reconfigThread) before starting again
4. Treat the exception as an idempotency signal: catch it, log, and skip, since a reload is already in progress

Example fix

// before
server.startReconfigurationTask(); // throws IOException if one is already running

// after
ReconfigurationTaskStatus s = server.getReconfigurationTaskStatus();
boolean running = s.getStartTime() > 0 && s.getEndTime() == 0;
if (!running) {
  server.startReconfigurationTask();
} else {
  LOG.info("Reconfiguration already in progress; skipping.");
}

// ReconfigurableBase reports endTime==0 while a task is running
ReconfigurationTaskStatus s = server.getReconfigurationTaskStatus();
if (s.getStartTime() > 0 && s.getEndTime() == 0) {
  // a reconfiguration task is in progress; do NOT call startReconfigurationTask()
}

try {
  server.startReconfigurationTask();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Another reconfiguration task")) {
    LOG.info("Reconfiguration already running; skipping duplicate trigger.");
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling startReconfigurationTask() again before the first task completed and before shutdownReconfigurationTask() cleared it; repeated admin 'apply config' RPCs (NameNode/DataNode/KMS style) where a client retries or double-submits while the first reload is still executing.

Common situations: Double-clicking a config-apply button or re-running a refresh script; automation/monitoring that re-triggers reconfiguration on timeout; a slow reconfigurePropertyImpl (e.g. a property whose apply does I/O) widening the window in which a second call lands.

Related errors

• Could not change property ${property} from '${oldVal}' to '$
• Not a valid Boolean value for {property} in reconfSlowPeerPa
• Not a valid Boolean value for {property}
• No directory is specified.
• No more entry in " + f

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b1fea27b6de8cae6. Report an issue: GitHub.