apache/hadoop · error · IOException

Unexpected format of JMX JSON response for: {}

Error message

Unexpected format of JMX JSON response for: {}

What it means

DynoInfraUtils polls a Hadoop daemon's JMX servlet (e.g. http://<host>:<httpPort>/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo) with an HTTP GET and parses the body as a strict Jackson token stream that must begin with {"beans":[{ . If the HTTP status is 200 but the token sequence does not match START_OBJECT -> FIELD_NAME 'beans' -> START_ARRAY -> START_OBJECT, this IOException is thrown. It means the endpoint answered, but the payload is not a JMX beans document.

Source

Thrown at hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/DynoInfraUtils.java:564

          nnWebUri.getPort(), "/jmx?qry=" + jmxBeanQuery);
    } catch (MalformedURLException e) {
      throw new IllegalArgumentException("Invalid JMX query: \"" + jmxBeanQuery
          + "\" against " + "NameNode URI: " + nnWebUri);
    }
    HttpURLConnection conn = (HttpURLConnection) queryURL.openConnection();
    if (conn.getResponseCode() != 200) {
      throw new IOException(
          "Unable to retrieve JMX: " + conn.getResponseMessage());
    }
    InputStream in = conn.getInputStream();
    JsonFactory fac = new JsonFactory();
    JsonParser parser = fac.createParser(in);
    if (parser.nextToken() != JsonToken.START_OBJECT
        || parser.nextToken() != JsonToken.FIELD_NAME
        || !parser.getCurrentName().equals("beans")
        || parser.nextToken() != JsonToken.START_ARRAY
        || parser.nextToken() != JsonToken.START_OBJECT) {
      throw new IOException(
          "Unexpected format of JMX JSON response for: " + jmxBeanQuery);
    }
    int objectDepth = 1;
    String ret = null;
    while (objectDepth > 0) {
      JsonToken tok = parser.nextToken();
      if (tok == JsonToken.START_OBJECT) {
        objectDepth++;
      } else if (tok == JsonToken.END_OBJECT) {
        objectDepth--;
      } else if (tok == JsonToken.FIELD_NAME) {
        if (parser.getCurrentName().equals(property)) {
          parser.nextToken();
          ret = parser.getText();
          break;
        }
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

  1. curl the exact jmxBeanQuery URL shown in the message and confirm the body starts with {"beans":[{
  2. Point the query at the daemon's HTTP server port (dfs.namenode.http-address / dfs.datanode.http-address), not the RPC port
  3. Check that the qry= value matches the MBean exactly, e.g. Hadoop:service=NameNode,name=NameNodeInfo
  4. If polling during startup, retry with backoff until the bean is registered instead of failing immediately

Example fix

// before: JMX URL pointed at the NameNode RPC port
String url = "http://nn1:8020/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo";
// after: use the HTTP server port and verify shape first
String url = "http://nn1:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo";
// curl "$url" must print a body starting with {"beans":[{
Defensive patterns

Strategy: validation

Validate before calling

// pre-flight: confirm the endpoint returns a beans document before polling
try (InputStream in = new URL(jmxUrl).openStream()) {
  JsonParser p = new JsonFactory().createParser(in);
  boolean ok = p.nextToken() == JsonToken.START_OBJECT
      && p.nextToken() == JsonToken.FIELD_NAME && "beans".equals(p.getCurrentName())
      && p.nextToken() == JsonToken.START_ARRAY
      && p.nextToken() == JsonToken.START_OBJECT;
  if (!ok) throw new IllegalStateException("endpoint does not serve JMX beans: " + jmxUrl);
}

Try / catch

catch (IOException e) { /* log jmxBeanQuery; sleep; retry with backoff until startup timeout, since a daemon mid-boot may briefly serve odd 200 payloads */ }

Prevention

When it happens

Trigger: A 200 response whose body is not the expected JMX shape: the URL points at the RPC port or a UI page instead of the JMX servlet, an authentication filter or proxy returns a 200 HTML login/error page, or the qry= bean filter matches zero beans so the document is {"beans":[]}.

Common situations: Using the NameNode RPC port (8020/9000) instead of the HTTP port (9870/50070), a gateway/SSL proxy in front of the daemon rewriting responses, polling during daemon startup before JMX beans register, or a typo in the qry= MBean name.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5aaa9e937f478495. Report an issue: GitHub.