{"record":{"id":"cb045c7266e3d13b","repo":"apache/hadoop","slug":"content-length-is-missing-headers","errorCode":null,"errorMessage":"Content-Length is missing: ${headers}","messagePattern":"Content-Length is missing: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java","lineNumber":153,"sourceCode":"    // Use the original url if no resolved url exists, eg. if\n    // it's the first time a request is made.\n    final boolean resolved = resolvedURL.getURL() != null;\n    final URLOpener opener = resolved? resolvedURL: originalURL;\n\n    final HttpURLConnection connection = opener.connect(startOffset, resolved);\n    resolvedURL.setURL(getResolvedUrl(connection));\n\n    InputStream in = connection.getInputStream();\n    final Long length;\n    final Map<String, List<String>> headers = connection.getHeaderFields();\n    if (isChunkedTransferEncoding(headers)) {\n      // file length is not known\n      length = null;\n    } else {\n      // for non-chunked transfer-encoding, get content-length\n      final String cl = connection.getHeaderField(HttpHeaders.CONTENT_LENGTH);\n      if (cl == null) {\n        throw new IOException(HttpHeaders.CONTENT_LENGTH + \" is missing: \"\n            + headers);\n      }\n      final long streamlength = Long.parseLong(cl);\n      length = startOffset + streamlength;\n\n      // Java has a bug with >2GB request streams.  It won't bounds check\n      // the reads so the transfer blocks until the server times out\n      in = new BoundedInputStream(in, streamlength);\n    }\n\n    return new InputStreamAndFileLength(length, in);\n  }\n\n  private static boolean isChunkedTransferEncoding(\n      final Map<String, List<String>> headers) {\n    return contains(headers, HttpHeaders.TRANSFER_ENCODING, \"chunked\")\n        || contains(headers, HttpHeaders.TE, \"chunked\");\n  }","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java#L135-L171","documentation":"When ByteRangeInputStream opens a ranged HTTP read (WebHDFS OPEN), it computes the file length from the response: chunked responses have unknown length, everything else must carry a Content-Length header. If the response is neither chunked nor has Content-Length, this IOException listing the full header map is thrown. Well-behaved WebHDFS servers always set Content-Length, so a missing one almost always means an intermediary (proxy, gateway) or a non-HDFS endpoint mangled the response.","triggerScenarios":"Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java:153 when the library encounters an invalid state.","commonSituations":"Corporate proxies or API gateways (Knox, nginx, custom LBs) between client and NameNode/DataNode HTTP ports that buffer or rewrite responses; transparent proxy inject error bodies without length; misconfigured http.proxyHost JVM properties routing WebHDFS traffic through a proxy.","solutions":["Check the headers printed in the message: a non-HDFS response (proxy error page, auth challenge) is visible there","Bypass the intermediary: connect the client directly to the NameNode/DataNode HTTP port, or fix the proxy to pass Content-Length through unbuffered","Clear stray JVM proxy settings (-Dhttp.proxyHost/-Dhttp.proxyPort, http.nonProxyHosts) so WebHDFS traffic is not proxied","Reproduce with curl -v on the same OPEN URL to confirm which hop drops the header"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-flight: confirm the endpoint answers like WebHDFS before streaming\nHttpURLConnection c = (HttpURLConnection) openUrl.openConnection();\nc.setRequestMethod(\"GET\");\nboolean chunked = c.getHeaderField(\"Transfer-Encoding\") != null;\nif (c.getHeaderField(\"Content-Length\") == null && !chunked) {\n  throw new IOException(\"endpoint \" + openUrl + \" omits Content-Length; proxy or wrong URL?\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  in = fs.open(path);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Content-Length is missing\")) {\n    // headers are in the message: inspect for proxy error pages, then bypass the intermediary\n    LOG.error(\"WebHDFS response lacked Content-Length; check proxies on path to NN/DN: {}\", e.getMessage());\n  }\n  throw e;\n}","preventionTips":["Point WebHDFS clients directly at NN/DN HTTP ports; do not route data traffic through generic proxies","Audit -Dhttp.proxyHost / global proxy JVM properties in client environments","Reproduce header loss with curl -v before blaming the client"],"tags":["hdfs","webhdfs","http","proxy","headers"],"backgroundTag":"missing-content-length-header","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}