apache/beam · error

'items' missing from Elasticsearch response

Error message

'items' missing from Elasticsearch response: {}

What it means

In ElasticsearchIO.Write's createWriteReport, the bulk response JSON has no 'items' array even though the write was expected to produce per-item results. The library logs a warning and reports the whole response as the error since it cannot attribute failures to individual documents. This is only expected when the bulk request never reached a functioning bulk endpoint, e.g. connectivity or proxy issues.

Solutions

1. Verify the Elasticsearch connection address and that POST /_bulk is actually reaching the cluster (curl the endpoint).
2. Check that authentication is correct and no proxy/gateway is intercepting responses.
3. Inspect the logged searchResult body to see what the server actually returned.
4. Enable ElasticsearchIO connection/test configuration (withConnectTimeout, healthcheck) and retry the write.

Example fix

// before: misconfigured host behind proxy
.withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration.create(hosts, "https://lb.example.com", "9200"))
// after: point directly at ES nodes reachable for _bulk
.withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration.create(hosts, "https://es-node1.example.com", "9200"))

// before the pipeline, verify _bulk reachability
curl -s -X POST 'https://es-host:9200/_bulk' -H 'Content-Type: application/x-ndjson' --data-binary '{"index":{}}\n{"a":1}\n' | grep -q '"items"' && echo OK || echo BAD

When it happens

Trigger: POST /_bulk returns 2xx but the body lacks an 'items' node (items.isMissingNode() or size == 0), such as when a proxy/CDN intercepts the request, wrong port serving HTML error pages, or the cluster returns a bare error object with HTTP 200-style body.

Common situations: Misconfigured Elasticsearch host/port pointing to a load balancer or login page; a reverse proxy swallowing the bulk API; Elasticsearch behind an auth gateway returning an error JSON without items; network blips mid-bulk.

Related errors

• Could not extract error codes from responseEntity
• 2xx codes should not be exceptions. Got status code
• Artifact not found at
• Can't load the client certificate from the keystore
• Cannot get Elasticsearch version

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/96b3e2da7c7656d5. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java:273

  static JsonNode parseResponse(HttpEntity responseEntity) throws IOException {
    return mapper.readValue(responseEntity.getContent(), JsonNode.class);
  }

  static List<Document> createWriteReport(
      HttpEntity responseEntity, @Nullable Set<String> allowedErrorTypes, boolean throwWriteErrors)
      throws IOException {

    List<Document> responses = new ArrayList<>();
    int numErrors = 0;
    JsonNode searchResult = parseResponse(responseEntity);
    StringBuilder errorMessages =
        new StringBuilder("Error writing to Elasticsearch, some elements could not be inserted:");
    JsonNode items = searchResult.path("items");

    if (items.isMissingNode() || items.size() == 0) {
      // This would only be expected in cases like connectivity issues or similar
      errorMessages.append(searchResult);
      LOG.warn("'items' missing from Elasticsearch response: {}", errorMessages);
    }

    // some items present in bulk might have errors, concatenate error messages and record
    // which items had errors
    for (JsonNode item : items) {
      Document result = Document.create().withResponseItemJson(item.toString());

      JsonNode error = item.findValue("error");
      if (error != null) {
        // N.B. An empty-string within the allowedErrorTypes Set implies all errors are allowed.
        String type = error.path("type").asText();
        String reason = error.path("reason").asText();
        String docId = item.findValue("_id").asText();
        JsonNode causedBy = error.path("caused_by"); // May not be present
        String cbReason = causedBy.path("reason").asText();
        String cbType = causedBy.path("type").asText();

        if (allowedErrorTypes == null

View on GitHub (pinned to 12126d8942)