apache/hadoop · info · IOException

Block move cancelled.

Error message

Block move cancelled.

What it means

Raised inside Dispatcher.dispatch when a queued block move is about to be sent but its source StorageGroup is already flagged iteration-over - i.e. dfs.balancer.max-iteration-time elapsed and this iteration was cancelled. It is a controlled cancellation, not a fault: the move is skipped and counted as failed for this iteration, and the info log right above states the reason. The balancer exits normally afterwards.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java:371

      final DDatanode dn = g.getDDatanode();
      if (dn.addPendingBlock(this)) {
        proxySource = dn;
        return true;
      }
      return false;
    }

    /** Dispatch the move to the proxy source & wait for the response. */
    private void dispatch() {
      Socket sock = new Socket();
      DataOutputStream out = null;
      DataInputStream in = null;
      try {
        if (source.isIterationOver()){
          LOG.info("Cancel moving " + this +
              " as iteration is already cancelled due to" +
              " dfs.balancer.max-iteration-time is passed.");
          throw new IOException("Block move cancelled.");
        }
        LOG.info("Start moving " + this);
        assert !(reportedBlock instanceof DBlockStriped);

        ExtendedBlock eb = new ExtendedBlock(nnc.getBlockpoolID(),
            reportedBlock.getBlock());
        final KeyManager km = nnc.getKeyManager();
        Token<BlockTokenIdentifier> accessToken = null;
        OutputStream unbufOut;
        InputStream unbufIn;
        int encryptionKeyRetryCount = 0;
        while (true) {
          try {
            accessToken = km.getAccessToken(eb,
                new StorageType[]{target.storageType}, new String[0]);
            sock.connect(
                NetUtils.createSocketAddr(target.getDatanodeInfo().
                    getXferAddr(Dispatcher.this.connectToDnViaHostname)),

View on GitHub (pinned to 2add963021)

Solutions

1. Treat as expected if you set a short dfs.balancer.max-iteration-time; run the balancer again - remaining blocks are handled in the next run
2. Increase dfs.balancer.max-iteration-time (ms) so an iteration can drain its move queue
3. Reduce per-iteration work (lower dfs.datanode.balance.max.concurrent.moves or a smaller -threshold) so fewer moves are pending when time expires

Example fix

# before
<property><name>dfs.balancer.max-iteration-time</name><value>300000</value></property>

# after
<property><name>dfs.balancer.max-iteration-time</name><value>1800000</value></property>

catch (IOException e) {
  if (e.getMessage().contains("Block move cancelled")) { LOG.debug("Iteration deadline hit; move will be retried next run"); }
  else { throw e; }
}

When it happens

Trigger: Setting dfs.balancer.max-iteration-time to a small value (or a genuinely long iteration) so that moves still in the queue when time expires hit source.isIterationOver() == true.

Common situations: Clusters that cap balancer iterations to bound impact during business hours; very large rebalances where the dispatch queue outlives the iteration window.

Related errors

• Block move timed out
• currentKey hasn't been initialized.
• Can't load state from image in a running SecretManager.
• Can't add persisted delegation token to a running SecretMana
• Can't update persisted delegation token renewal to a running

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef578898fa0745f6. Report an issue: GitHub.