apache/hadoop · error · IOException

Failed to rename %s to %s, %s is a file

Error message

Failed to rename %s to %s, %s is a file

What it means

ZombieCluster.buildCluster() walks the LoggedNetworkTopology tree and requires every leaf to sit at the same depth; the first time it finds a leaf at a different depth than an earlier leaf it throws IllegalArgumentException. Depth matters because rumen hard-codes the bottom two levels as RackNode -> MachineNode (see 'path[level-1].addChild(current)' and the RackNode/MachineNode split in the second pass), so an unbalanced tree would misassign machine/rack roles. This is a data-quality contract on the topology JSON.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java:776

    FileStatus dstStatus;
    try {
      dstStatus = getFileStatus(dstPath);
    } catch (FileNotFoundException fnde) {
      dstStatus = null;
    }

    if (dstStatus == null) {
      Path dstParent = dstPath.getParent();
      if (dstParent != null) {
        Path currentPath = dstParent;
        while (currentPath != null
            && !currentPath.isRoot()) {
          FileStatus ancestorStatus;
          try {
            ancestorStatus = getFileStatus(currentPath);
            if (!ancestorStatus.isDirectory()) {
              throw new IOException(String.format(
                  "Failed to rename %s to %s, %s is a file",
                  srcPath, dstPath, currentPath));
            }
            break;
          } catch (FileNotFoundException fnde) {
            currentPath = currentPath.getParent();
          }
        }
      }
      LOG.debug("Parent directory {} does not exist "
          + "or will be implicitly created during rename",
          dstPath.getParent());
    } else {
      if (dstStatus.isDirectory()) {
        dstPath = new Path(dstPath, srcPath.getName());
        FileStatus status;
        try {
          status = getFileStatus(dstPath);

View on GitHub (pinned to 2add963021)

Solutions

  1. Normalize the topology JSON so every path from root to leaf has the same number of levels (standard: root -> rack -> host).
  2. Locate the imbalance by printing each leaf's depth from the LoggedNetworkTopology tree before constructing ZombieCluster (a 10-line DFS makes the offending branch obvious).
  3. Remove empty/placeholder children entries — an entry with no children is a leaf and any depth mismatch counts.
  4. For genuinely deeper topologies, flatten intermediate switches into rack names (rack/switch naming) to keep the two-level rack/host model.

Example fix

// before (topology.json): unbalanced
// { "name":"/dc", "children":[
//   { "name":"/r1", "children":[ {"name":"h1"} ] },
//   { "name":"/r2", "children":[
//     { "name":"/sw", "children":[ {"name":"h2"} ] } ] } ] }
new ZombieCluster(topology, defaultNode); // IllegalArgumentException

// after (topology.json): balanced root -> rack -> host
// { "name":"/dc", "children":[
//   { "name":"/r1", "children":[ {"name":"h1"} ] },
//   { "name":"/r2", "children":[ {"name":"h2"} ] } ] }
new ZombieCluster(topology, defaultNode);
Defensive patterns

Strategy: validation

Validate before calling

// Verify all leaves share one depth BEFORE building the ZombieCluster
static int checkBalanced(LoggedNetworkTopology n, int depth, Integer leafDepth,
    java.util.List<String> badLeaves) {
  List<LoggedNetworkTopology> kids = n.getChildren();
  if (kids == null || kids.isEmpty()) {
    if (leafDepth != null && leafDepth != depth) badLeaves.add(n.getName().getValue());
    return leafDepth == null ? depth : leafDepth;
  }
  for (LoggedNetworkTopology c : kids) leafDepth = checkBalanced(c, depth + 1, leafDepth, badLeaves);
  return leafDepth;
}
// if (!badLeaves.isEmpty()) fail with the offending leaf names

Try / catch

try {
  cluster = new ZombieCluster(topology, defaultNode);
} catch (IllegalArgumentException e) {
  if ("Leaf nodes are not on the same level".equals(e.getMessage())) {
    throw new IOException("Topology file has unbalanced depth; normalize to root/rack/host", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: new ZombieCluster(topology, defaultNode) where one branch of the topology ends at depth 1 and another at depth 2 — e.g. one rack entry has bare host children while another has an intermediate switch layer; feeding ClusterTopologyReader a topology file with a stray empty node; hand-authored topology JSON with inconsistent nesting.

Common situations: Editing a rumen topology JSON by hand and forgetting a level; combining topology dumps from different cluster generations; clusters modeled with per-rack switch hierarchies only in part of the datacenter.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/284d7b39eebbe61c. Report an issue: GitHub.