apache/hadoop · info · IOException

Block move cancelled.

Error message

Block move cancelled.

What it means

Raised inside Dispatcher.dispatch when a queued block move is about to be sent but its source StorageGroup is already flagged iteration-over - i.e. dfs.balancer.max-iteration-time elapsed and this iteration was cancelled. It is a controlled cancellation, not a fault: the move is skipped and counted as failed for this iteration, and the info log right above states the reason. The balancer exits normally afterwards.

Solutions

1. Treat as expected if you set a short dfs.balancer.max-iteration-time; run the balancer again - remaining blocks are handled in the next run
2. Increase dfs.balancer.max-iteration-time (ms) so an iteration can drain its move queue
3. Reduce per-iteration work (lower dfs.datanode.balance.max.concurrent.moves or a smaller -threshold) so fewer moves are pending when time expires

Example fix

# before
<property><name>dfs.balancer.max-iteration-time</name><value>300000</value></property>

# after
<property><name>dfs.balancer.max-iteration-time</name><value>1800000</value></property>

catch (IOException e) {
  if (e.getMessage().contains("Block move cancelled")) { LOG.debug("Iteration deadline hit; move will be retried next run"); }
  else { throw e; }
}

When it happens

Trigger: Setting dfs.balancer.max-iteration-time to a small value (or a genuinely long iteration) so that moves still in the queue when time expires hit source.isIterationOver() == true.

Common situations: Clusters that cap balancer iterations to bound impact during business hours; very large rebalances where the dispatch queue outlives the iteration window.

Related errors

• Block move timed out
• Another is running.
• args =
• Balancer without BlockPlacementPolicyDefault
• Can't add persisted delegation token to a running…

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef578898fa0745f6. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java:371

      final DDatanode dn = g.getDDatanode();
      if (dn.addPendingBlock(this)) {
        proxySource = dn;
        return true;
      }
      return false;
    }

    /** Dispatch the move to the proxy source & wait for the response. */
    private void dispatch() {
      Socket sock = new Socket();
      DataOutputStream out = null;
      DataInputStream in = null;
      try {
        if (source.isIterationOver()){
          LOG.info("Cancel moving " + this +
              " as iteration is already cancelled due to" +
              " dfs.balancer.max-iteration-time is passed.");
          throw new IOException("Block move cancelled.");
        }
        LOG.info("Start moving " + this);
        assert !(reportedBlock instanceof DBlockStriped);

        ExtendedBlock eb = new ExtendedBlock(nnc.getBlockpoolID(),
            reportedBlock.getBlock());
        final KeyManager km = nnc.getKeyManager();
        Token<BlockTokenIdentifier> accessToken = null;
        OutputStream unbufOut;
        InputStream unbufIn;
        int encryptionKeyRetryCount = 0;
        while (true) {
          try {
            accessToken = km.getAccessToken(eb,
                new StorageType[]{target.storageType}, new String[0]);
            sock.connect(
                NetUtils.createSocketAddr(target.getDatanodeInfo().
                    getXferAddr(Dispatcher.this.connectToDnViaHostname)),

View on GitHub (pinned to 2add963021)