{"record":{"id":"dc71abd49499af6f","repo":"apache/hadoop","slug":"failed-update-mount-table-mount","errorCode":null,"errorMessage":"Failed update mount table \" + mount","messagePattern":"Failed update mount table \" \\+ mount","errorType":"exception","errorClass":"java.io.IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/rbfbalance/MountTableProcedure.java","lineNumber":120,"sourceCode":"        RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);\n    InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);\n    RouterClient rClient = new RouterClient(routerSocket, conf);\n    try {\n      MountTableManager mountTable = rClient.getMountTableManager();\n\n      MountTable originalEntry = getMountEntry(mount, mountTable);\n      if (originalEntry == null) {\n        throw new IOException(\"Mount table \" + mount + \" doesn't exist\");\n      } else {\n        RemoteLocation remoteLocation =\n            new RemoteLocation(dstNs, dstPath, mount);\n        originalEntry.setDestinations(Arrays.asList(remoteLocation));\n        UpdateMountTableEntryRequest updateRequest =\n            UpdateMountTableEntryRequest.newInstance(originalEntry);\n        UpdateMountTableEntryResponse response =\n            mountTable.updateMountTableEntry(updateRequest);\n        if (!response.getStatus()) {\n          throw new IOException(\"Failed update mount table \" + mount);\n        }\n        rClient.getMountTableManager().refreshMountTableEntries(\n            RefreshMountTableEntriesRequest.newInstance());\n      }\n    } finally {\n      rClient.close();\n    }\n  }\n\n  /**\n   * Gets the mount table entry.\n   * @param mount name of the mount entry.\n   * @param mountTable the mount table.\n   * @return corresponding mount entry.\n   * @throws IOException in case of failure to retrieve mount entry.\n   */\n  public static MountTable getMountEntry(String mount,\n      MountTableManager mountTable)","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/rbfbalance/MountTableProcedure.java#L102-L138","documentation":"The FedBalance switch step failed because the router's MountTableManager.updateMountTableEntry() RPC returned a failure status (response.getStatus() == false) instead of throwing. updateMountTableDestination() had already fetched and modified the entry (destination replaced by dstNs/dstPath) and treats the rejected update as a hard IOException, aborting the balance job before the mount table refresh.","triggerScenarios":"The router accepts the connection but the update fails server-side: state store (ZooKeeper/DB) write error or session loss, the mount record changed concurrently, router internal error while persisting the entry.","commonSituations":"ZooKeeper session expiry or state store hiccup exactly during the switch phase; a second admin operation editing the same mount entry at once; router restarting while the balance job hits its switch step.","solutions":["Check the router and state store (ZooKeeper/DB) logs at the failure timestamp for the underlying write error","Verify the mount entry's current state with hdfs dfsrouteradmin -list","Once the state store is healthy, retry the fedbalance job — the procedure re-executes the switch phase","Serialize admin changes to the same mount point so nothing races the procedure"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Retry the switch step with backoff; it is re-entrant after state store recovery\nfor (int attempt = 1; attempt <= 3; attempt++) {\n  try {\n    MountTableProcedure.updateMountTableDestination(mount, dstNs, dstPath, conf);\n    break;\n  } catch (IOException e) {\n    if (attempt == 3 || !isTransient(e)) throw e;\n    TimeUnit.SECONDS.sleep(5L * attempt);\n  }\n}","preventionTips":["Keep the state store quorum healthy for the whole duration of balance jobs","Avoid concurrent dfsrouteradmin edits to the mount being migrated","Alert on router admin update failures so switch phases are retried promptly"],"tags":["hadoop","rbf","fedbalance","mount-table","state-store","update-failed"],"backgroundTag":"admin-api-update-rejected","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}