apache/hadoop · error · IOException

Federated metrics is not initialized

Error message

Federated metrics is not initialized

What it means

Thrown by NamenodeBeanMetrics.getRBFMetrics() when the Router's RBFMetrics object is null, meaning the Router has not registered its federation metrics (JMX) yet. The Router only creates RBFMetrics when its metrics service (RouterMetricsService) initializes, so this indicates the Router is not fully started, is stopping, or was constructed without the metrics service. It is an IOException surfacing through the NameNodeMXBean facade that proxies federated metrics to JMX clients.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/NamenodeBeanMetrics.java:178

    if (fsStateBeanName != null) {
      MBeans.unregister(fsStateBeanName);
      fsStateBeanName = null;
    }
    if (nnInfoBeanName != null) {
      MBeans.unregister(nnInfoBeanName);
      nnInfoBeanName = null;
    }
    // Remove the NameNode status bean
    if (nnStatusBeanName != null) {
      MBeans.unregister(nnStatusBeanName);
      nnStatusBeanName = null;
    }
  }

  private RBFMetrics getRBFMetrics() throws IOException {
    RBFMetrics metrics = getRouter().getMetrics();
    if (metrics == null) {
      throw new IOException("Federated metrics is not initialized");
    }
    return metrics;
  }

  /////////////////////////////////////////////////////////
  // NameNodeMXBean
  /////////////////////////////////////////////////////////

  @Override
  public String getVersion() {
    return VersionInfo.getVersion() + ", r" + VersionInfo.getRevision();
  }

  @Override
  public String getSoftwareVersion() {
    return VersionInfo.getVersion();
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for full Router startup (log line 'Router: STARTED' / serviceStart completed) before querying the NameNode JMX beans
2. Verify the Router process is healthy: check state store connectivity and the router.out/log for failed service init
3. In tests, mock Router.getMetrics() to return a non-null RBFMetrics before exercising NamenodeBeanMetrics
4. Retry the JMX/RPC query on a short interval; transient during restarts

// Check the Router is in STARTED state before querying federation metrics
if (router.getServiceStatus() != Service.STATE.STARTED || router.getMetrics() == null) {
  throw new IllegalStateException("Router metrics not ready; retry after startup");
}
NamenodeBeanMetrics nnMetrics = router.getNamenodeMetrics();

try {
  RBFMetrics m = nnBeanMetrics.getRBFMetricsSource(); // wrapped getter
} catch (IOException e) {
  if (e.getMessage().contains("Federated metrics is not initialized")) {
    // startup/shutdown window: back off and retry, do not treat as data error
  } else { throw e; }
}

When it happens

Trigger: Calling any NameNodeMXBean method on the Router (e.g. JMX query on NamenodeBeanMetrics, getRouter()/getMembers()/getNameservices() MBean attributes) before Router.serviceStart() completes or after serviceStop(); unit tests instantiating NamenodeBeanMetrics with a Router stub whose getMetrics() returns null; a Router whose state store connection failed during init so metrics were never bound.

Common situations: Monitoring tools (Prometheus JMX exporter, Grafana) scraping the Router's JMX port during Router startup/restart; test harnesses that mock Router without stubbing getMetrics(); Router startup races where the admin RPC service is up but the metrics service is not yet registered.

Related errors

• Namenode metrics is not initialized
• Cannot serialize field {} into JSON
• Cached State Store not initialized, {recordClass} records no
• cannot locate OfferService thread for bp={blockPoolId}
• {} is not boolean value

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/36621728289f9829. Report an issue: GitHub.