{"record":{"id":"96b3e2da7c7656d5","repo":"apache/beam","slug":"items-missing-from-elasticsearch-response","errorCode":null,"errorMessage":"'items' missing from Elasticsearch response: {}","messagePattern":"'items' missing from Elasticsearch response: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java","lineNumber":273,"sourceCode":"  static JsonNode parseResponse(HttpEntity responseEntity) throws IOException {\n    return mapper.readValue(responseEntity.getContent(), JsonNode.class);\n  }\n\n  static List<Document> createWriteReport(\n      HttpEntity responseEntity, @Nullable Set<String> allowedErrorTypes, boolean throwWriteErrors)\n      throws IOException {\n\n    List<Document> responses = new ArrayList<>();\n    int numErrors = 0;\n    JsonNode searchResult = parseResponse(responseEntity);\n    StringBuilder errorMessages =\n        new StringBuilder(\"Error writing to Elasticsearch, some elements could not be inserted:\");\n    JsonNode items = searchResult.path(\"items\");\n\n    if (items.isMissingNode() || items.size() == 0) {\n      // This would only be expected in cases like connectivity issues or similar\n      errorMessages.append(searchResult);\n      LOG.warn(\"'items' missing from Elasticsearch response: {}\", errorMessages);\n    }\n\n    // some items present in bulk might have errors, concatenate error messages and record\n    // which items had errors\n    for (JsonNode item : items) {\n      Document result = Document.create().withResponseItemJson(item.toString());\n\n      JsonNode error = item.findValue(\"error\");\n      if (error != null) {\n        // N.B. An empty-string within the allowedErrorTypes Set implies all errors are allowed.\n        String type = error.path(\"type\").asText();\n        String reason = error.path(\"reason\").asText();\n        String docId = item.findValue(\"_id\").asText();\n        JsonNode causedBy = error.path(\"caused_by\"); // May not be present\n        String cbReason = causedBy.path(\"reason\").asText();\n        String cbType = causedBy.path(\"type\").asText();\n\n        if (allowedErrorTypes == null","sourceCodeStart":255,"sourceCodeEnd":291,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L255-L291","documentation":"In ElasticsearchIO.Write's createWriteReport, the bulk response JSON has no 'items' array even though the write was expected to produce per-item results. The library logs a warning and reports the whole response as the error since it cannot attribute failures to individual documents. This is only expected when the bulk request never reached a functioning bulk endpoint, e.g. connectivity or proxy issues.","triggerScenarios":"POST /_bulk returns 2xx but the body lacks an 'items' node (items.isMissingNode() or size == 0), such as when a proxy/CDN intercepts the request, wrong port serving HTML error pages, or the cluster returns a bare error object with HTTP 200-style body.","commonSituations":"Misconfigured Elasticsearch host/port pointing to a load balancer or login page; a reverse proxy swallowing the bulk API; Elasticsearch behind an auth gateway returning an error JSON without items; network blips mid-bulk.","solutions":["Verify the Elasticsearch connection address and that POST /_bulk is actually reaching the cluster (curl the endpoint).","Check that authentication is correct and no proxy/gateway is intercepting responses.","Inspect the logged searchResult body to see what the server actually returned.","Enable ElasticsearchIO connection/test configuration (withConnectTimeout, healthcheck) and retry the write."],"exampleFix":"// before: misconfigured host behind proxy\n.withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration.create(hosts, \"https://lb.example.com\", \"9200\"))\n// after: point directly at ES nodes reachable for _bulk\n.withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration.create(hosts, \"https://es-node1.example.com\", \"9200\"))","handlingStrategy":"validation","validationCode":"// before the pipeline, verify _bulk reachability\ncurl -s -X POST 'https://es-host:9200/_bulk' -H 'Content-Type: application/x-ndjson' --data-binary '{\"index\":{}}\\n{\"a\":1}\\n' | grep -q '\"items\"' && echo OK || echo BAD","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Point ConnectionConfiguration directly at real Elasticsearch nodes, not a proxy/login page.","Verify credentials and that no auth gateway intercepts _bulk responses.","Run ElasticsearchIO's read/test connections before writing."],"tags":["elasticsearch","http","malformed-response"],"backgroundTag":"unexpected-response-shape","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}