What causes "The ${useErrorStream ? "error" : "input"} stream is null." in apache/hadoop?

Reproduce the request with curl -i against the exact WebHDFS URL and inspect status, Content-Length, and body.

apache/hadoop · error · IOException

The ${useErrorStream ? "error" : "input"} stream is null.

Error message

The ${useErrorStream ? "error" : "input"} stream is null.

What it means

WebHdfsFileSystem.jsonParse chooses either the HTTP error stream or input stream and refuses to parse a null stream. A zero content length returns null earlier, so this IOException means the connection advertised a nonzero or unknown content length but HttpURLConnection could not provide the corresponding body. The WebHDFS response therefore cannot be decoded.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java:491

          result);
    }
    workingDir = absolutePath;
  }

  private Path makeAbsolute(Path f) {
    return f.isAbsolute()? f: new Path(workingDir, f);
  }

  @VisibleForTesting
  public static Map<?, ?> jsonParse(final HttpURLConnection c,
      final boolean useErrorStream) throws IOException {
    if (c.getContentLength() == 0) {
      return null;
    }
    final InputStream in = useErrorStream ?
        c.getErrorStream() : c.getInputStream();
    if (in == null) {
      throw new IOException("The " + (useErrorStream? "error": "input") +
          " stream is null.");
    }
    try {
      final String contentType = c.getContentType();
      if (contentType != null) {
        final MediaType parsed = MediaType.valueOf(contentType);
        if (!MediaType.APPLICATION_JSON_TYPE.isCompatible(parsed)) {
          throw new IOException("Content-Type \"" + contentType
              + "\" is incompatible with \"" + MediaType.APPLICATION_JSON
              + "\" (parsed=\"" + parsed + "\")");
        }
      }
      return JsonSerialization.mapReader().readValue(in);
    } finally {
      in.close();
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

Reproduce the request with curl -i against the exact WebHDFS URL and inspect status, Content-Length, and body.
Check NameNode, HttpFS, and reverse-proxy logs for the same request; this is a transport/response defect rather than bad application input.
Remove or fix proxies that strip or truncate response bodies and disable unsafe response buffering/rewriting for WebHDFS paths.
Retry once on a new connection to rule out a stale keep-alive socket, then report the endpoint failure if it repeats.

Example fix

// before
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
Map<?, ?> json = WebHdfsFileSystem.jsonParse(conn, false);

// after: diagnose at the HTTP boundary before JSON parsing
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
int code = conn.getResponseCode();
InputStream body = code >= 400 ? conn.getErrorStream() : conn.getInputStream();
if (body == null) {
  throw new IOException("WebHDFS endpoint returned no body for HTTP " + code);
}
Map<?, ?> json = WebHdfsFileSystem.jsonParse(conn, false);

Defensive patterns

Strategy: try-catch

Try / catch

try {
  return fs.getFileStatus(path);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().endsWith("stream is null.")) {
    LOG.error("WebHDFS endpoint returned headers without a body for {}", path, e);
    // One retry can recover a stale keep-alive connection; persistent failures are endpoint/proxy defects.
    return retryOnceOnNewFileSystem(fs.getUri(), fs.getConf(), path);
  }
  throw e;
}

Prevention

Use curl -i to verify that every WebHDFS request returns a body consistent with its Content-Length.
Do not deploy proxies that strip or buffer response bodies incorrectly.
Close and recreate FileSystem/connections after transport errors rather than reusing a suspect connection.

When it happens

Trigger: A WebHDFS operation reaches jsonParse and c.getContentLength() is not 0, but getInputStream() or getErrorStream() returns null. This occurs when a server or proxy closes the connection without a body, sends malformed HTTP, or supplies headers inconsistent with the actual response.

Common situations: A reverse proxy, firewall, or API gateway drops a WebHDFS response body; a NameNode or HttpFS process fails while writing the response; keep-alive connection reuse hands the client a stale connection; an overloaded server resets the connection.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ab3cac4bc2ea70d1. Report an issue: GitHub.