apache/hadoop · error · IOException

Another {name} is running.

Error message

Another {name} is running.

What it means

NameNodeConnector's constructor calls checkAndMarkRunning, which enforces a single balancer instance per namespace via a lock file (default /system/balancer.id in HDFS). If the file exists it first tries fs.append(idPath): append fails while another instance holds the file's HDFS lease, so a failed append means a live peer and the constructor throws IOException('Another Balancer is running.'). A stale file from a cleanly-exited or killed balancer is deleted and reused, so seeing this error means a lease is genuinely held (or recovering).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:214

    this.getBlocksToStandby = !conf.getBoolean(
        DFSConfigKeys.DFS_NAMENODE_GETBLOCKS_CHECK_OPERATION_KEY,
        DFSConfigKeys.DFS_NAMENODE_GETBLOCKS_CHECK_OPERATION_DEFAULT);
    this.config = conf;

    this.fs = (DistributedFileSystem)FileSystem.get(nameNodeUri, conf);

    final NamespaceInfo namespaceinfo = namenode.versionRequest();
    this.blockpoolID = namespaceinfo.getBlockPoolID();

    final FsServerDefaults defaults = fs.getServerDefaults(new Path("/"));
    this.keyManager = new KeyManager(blockpoolID, namenode,
        defaults.getEncryptDataTransfer(), conf);
    // if it is for test, we do not create the id file
    if (checkOtherInstanceRunning) {
      out = checkAndMarkRunning();
      if (out == null) {
        // Exit if there is another one running.
        throw new IOException("Another " + name + " is running.");
      }
    }
  }

  public NameNodeConnector(String name, URI nameNodeUri, String nsId,
                           Path idPath, List<Path> targetPaths,
                           Configuration conf, int maxNotChangedIterations)
      throws IOException {
    this(name, nameNodeUri, idPath, targetPaths, conf, maxNotChangedIterations);
    this.nsId = nsId;
  }

  public DistributedFileSystem getDistributedFileSystem() {
    return fs;
  }

  /** @return the block pool ID */
  public String getBlockpoolID() {

View on GitHub (pinned to 2add963021)

Solutions

1. Find and stop the other running instance (check 'jps' / process lists on all gateways for a Balancer process), then rerun
2. If runs are long, serialize them in your scheduler (lock/flock or a single cron owner) instead of relying on this error
3. If you are certain nothing runs and the lease is stale, wait for HDFS lease recovery, or delete /system/balancer.id with hdfs dfs -rm after double-checking no balancer is active

Example fix

# before: overlapping cron
*/30 * * * * hdfs balancer

# after: serialized run with a local lock
30 1 * * * flock -n /tmp/balancer.lock hdfs balancer

// best-effort pre-check (racy; the append-probe is authoritative)
FileSystem fs = FileSystem.get(conf);
if (fs.exists(new Path("/system/balancer.id"))) {
  try { IOUtils.closeStream(fs.append(new Path("/system/balancer.id"))); }
  catch (IOException leaseHeld) { throw new IllegalStateException("A balancer appears to be running"); }
}

try { nnc = new NameNodeConnector(name, uri, idPath, targetPaths, conf, maxIdle); }
catch (IOException e) {
  if (e.getMessage().startsWith("Another")) { LOG.warn("{} - skipping scheduled run", e.getMessage()); return EXIT_ALREADY_RUNNING; }
  throw e;
}

When it happens

Trigger: Starting a second 'hdfs balancer' against the same namespace while the first still runs - overlapping cron jobs, two operators on different gateways, or a wrapper script that spawns the balancer twice.

Common situations: Cron schedules that overlap when runs are long; an old balancer process left running on another gateway; immediately re-running the balancer after killing one process (lease not yet recovered on the NameNode).

Related errors

• Balancer without BlockPlacementPolicyDefault
• {key} = {v} <= 0
• Number out of range: threshold = {threshold}
• args = {args}
• Failed to read {type} node list from file: {filename}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/148e4a638d3c07cb. Report an issue: GitHub.