{"record":{"id":"dd9f66dcd222de69","repo":"apache/hadoop","slug":"mount-table-state-store-is-not-available-dd9f66","errorCode":null,"errorMessage":"Mount table state store is not available.","messagePattern":"Mount table state store is not available\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUpdateService.java","lineNumber":188,"sourceCode":"                HdfsConstants.QUOTA_DONT_SET, gQuota.getTypeQuota(t), t);\n        LOG.info(\"[Fix Quota] src={} dst={} type={} oldQuota={} newQuota={}\",\n            location.getSrc(), location, t, remoteQuota.getTypeQuota(t),\n            gQuota.getTypeQuota(t));\n      }\n    }\n  }\n\n  /**\n   * Get mount table store management interface.\n   * @return MountTableStore instance.\n   * @throws IOException\n   */\n  private MountTableStore getMountTableStore() throws IOException {\n    if (this.mountTableStore == null) {\n      this.mountTableStore = router.getStateStore().getRegisteredRecordStore(\n          MountTableStore.class);\n      if (this.mountTableStore == null) {\n        throw new IOException(\"Mount table state store is not available.\");\n      }\n    }\n    return this.mountTableStore;\n  }\n\n  /**\n   * Get all the existing mount tables.\n   * @return List of mount tables.\n   * @throws IOException\n   */\n  private List<MountTable> getMountTableEntries() throws IOException {\n    // scan mount tables from root path\n    GetMountTableEntriesRequest getRequest = GetMountTableEntriesRequest\n        .newInstance(\"/\");\n    GetMountTableEntriesResponse getResponse = getMountTableStore()\n        .getMountTableEntries(getRequest);\n    return getResponse.getEntries();\n  }","sourceCodeStart":170,"sourceCodeEnd":206,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUpdateService.java#L170-L206","documentation":"RouterQuotaUpdateService.getMountTableStore lazily fetches the MountTableStore record store from the router's State Store. If the State Store has no registered MountTableStore - because the state store is disabled, failed to initialize its driver, or the record store never registered - it throws this IOException, which aborts the periodic quota cache update cycle.","triggerScenarios":"The quota update service period fires (dfs.federation.router.quota.cache.update.interval) and its first touch of the mount table store finds router.getStateStore().getRegisteredRecordStore(MountTableStore.class) == null: state store driver (ZK/JDBC) never connected, state store marked unavailable, or MountTableStore registration skipped during store init.","commonSituations":"ZooKeeper down or unreachable when the router started; wrong dfs.federation.router.store.driver class or connection string; state store initialization failing silently earlier (check earlier router log lines); rolling upgrade where the store schema/driver version mismatches.","solutions":["Check router startup logs for State Store driver errors and fix connectivity (ZK quorum address, JDBC URL, credentials)","Verify dfs.federation.router.store.driver is set to a valid implementation (e.g. StateStoreZooKeeperImpl or the JDBC driver) and dfs.federation.router.store.serializer matches what the other routers use","Restart the router once the State Store backend is healthy so record stores (including MountTableStore) register; the error clears on the next update cycle","If multiple routers share the store, confirm at least one router can write and the store isn't in a failed/closed state"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Probe state store health before the quota cycle needs it\nStateStore store = router.getStateStore();\nif (!store.isRunning() || store.getRegisteredRecordStore(MountTableStore.class) == null) {\n  LOG.warn(\"State store/MountTableStore unavailable; skipping quota update this cycle\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  updateSchedule();\n} catch (IOException e) {\n  if (\"Mount table state store is not available.\".equals(e.getMessage())) {\n    // transient store outage: log, let the periodic service retry on the next interval\n    LOG.warn(\"Quota cache update deferred: state store unavailable\");\n    return;\n  }\n  throw e;\n}","preventionTips":["Monitor State Store (ZK/JDBC) availability with dedicated alerts; most quota cache failures trace to it","Keep the store driver, serializer, and connection settings identical across all routers","Restart routers only after the state store backend is confirmed healthy so record stores register cleanly"],"tags":["hdfs","router","federation","quota","state-store","zookeeper","availability"],"backgroundTag":"state-store-unavailable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}