apache/hadoop · warning · IOException

Block move timed out

Error message

Block move timed out

What it means

Thrown from Dispatcher.receiveResponse while polling block-move responses from the proxy source. stopWaitingForResponse returns true when the source iteration is over or (blockMoveTimeout > 0 and) the wait has exceeded dfs.balancer.block-move.timeout milliseconds; the balancer then stops waiting and raises IOException('Block move timed out'). Importantly the datanode-side copy may still complete - only the balancer's wait is abandoned, so the block can still be counted moved later.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java:492

    /** Check whether to continue waiting for response */
    private boolean stopWaitingForResponse(long startTime) {
      return source.isIterationOver() ||
          (blockMoveTimeout > 0 &&
          (Time.monotonicNow() - startTime > blockMoveTimeout));
    }

    /** Receive a reportedBlock copy response from the input stream */
    private void receiveResponse(DataInputStream in) throws IOException {
      long startTime = Time.monotonicNow();
      BlockOpResponseProto response =
          BlockOpResponseProto.parseFrom(vintPrefixed(in));
      while (response.getStatus() == Status.IN_PROGRESS) {
        // read intermediate responses
        response = BlockOpResponseProto.parseFrom(vintPrefixed(in));
        // Stop waiting for slow block moves. Even if it stops waiting,
        // the actual move may continue.
        if (stopWaitingForResponse(startTime)) {
          throw new IOException("Block move timed out");
        }
      }
      String logInfo = "reportedBlock move is failed";
      DataTransferProtoUtil.checkBlockOpStatus(response, logInfo, true);
    }

    /** reset the object */
    private void reset() {
      reportedBlock = null;
      source = null;
      proxySource = null;
      target = null;
    }
  }

  private static boolean prepareRetryAfterInvalidEncryptionKey(KeyManager km,
      int retryCount) throws IOException {
    if (retryCount > 1) {

View on GitHub (pinned to 2add963021)

Solutions

1. Increase dfs.balancer.block-move.timeout (ms), or set it back to 0 (default) to wait indefinitely, then rerun the balancer
2. Raise dfs.datanode.balance.bandwidthPerSec or lower dfs.datanode.balance.max.concurrent.moves so individual moves finish faster
3. Investigate chronically slow datanodes (disk health, network) named in the surrounding move log lines

Example fix

# before
<property><name>dfs.balancer.block-move.timeout</name><value>30000</value></property>

# after
<property><name>dfs.balancer.block-move.timeout</name><value>600000</value></property>

catch (IOException e) {
  if (e.getMessage().equals("Block move timed out")) { LOG.warn("Move wait abandoned; datanode copy may still finish"); return; }
  throw e;
}

When it happens

Trigger: dfs.balancer.block-move.timeout set to a positive value smaller than the real time a replaceBlock operation takes (slow disks, throttled bandwidth dfs.datanode.balance.bandwidthPerSec, huge blocks, or datanodes under load), or the iteration time expiring mid-wait.

Common situations: Operators adding a block-move timeout to stop the balancer 'hanging' on stalled datanodes; aggressive bandwidth limits making every large block exceed the timeout; shared clusters where target datanodes are busy.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Block move cancelled.
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes
• Timed out waiting for format() response
• Timed out waiting for response from loggers
• Timed out waiting for doPreUpgrade() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8c5f4fcd2dfacc63. Report an issue: GitHub.