apache/beam · error · IOException

<errorMessages.toString()>

Error message

<errorMessages.toString()>

What it means

WriteToElasticsearch batches bulk API requests and collects per-item error messages from Elasticsearch responses. If any bulk item reports an error and throwWriteErrors is enabled, the whole batch fails with an IOException containing all accumulated error messages. This surfaces partial bulk-write failures (mapping conflicts, invalid docs, cluster issues) as a single pipeline failure.

Source

Thrown at sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java:310

            || (!allowedErrorTypes.contains(type) && !allowedErrorTypes.contains(cbType))) {
          // 'error' and 'causedBy` fields are not null, and the error is not being ignored.
          result = result.withHasError(true);
          numErrors++;

          errorMessages.append(String.format("%nDocument id %s: %s (%s)", docId, reason, type));

          if (!causedBy.isMissingNode()) {
            errorMessages.append(String.format("%nCaused by: %s (%s)", cbReason, cbType));
          }
        }
      }
      responses.add(result);
    }

    if (numErrors > 0) {
      LOG.error("{}", errorMessages.toString());
      if (throwWriteErrors) {
        throw new IOException(errorMessages.toString());
      }
    }

    return responses;
  }

  /** A POJO describing a connection configuration to Elasticsearch. */
  @AutoValue
  public abstract static class ConnectionConfiguration implements Serializable {

    public abstract List<String> getAddresses();

    public abstract @Nullable String getUsername();

    public abstract @Nullable String getPassword();

    public abstract @Nullable String getApiKey();

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the aggregated errorMessages in the exception to find the failing documents and their ES error reasons
2. Fix the offending documents/field types or update the index mapping
3. If partial success is acceptable, call .setThrowWriteErrors(false) on Write.fromConnectionConfiguration so errors are only logged, not thrown
4. Check ES cluster health and logs for systemic issues causing item failures

Example fix

// before
Write write = ElasticsearchIO.Write.with(name, connection)
    .setThrowWriteErrors(true);
// after (log errors instead of failing the pipeline)
Write write = ElasticsearchIO.Write.with(name, connection)
    .setThrowWriteErrors(false);

// Cannot catch inside DoFn; decide policy at construction time
try {
  pipeline.apply(ElasticsearchIO.write().setThrowWriteErrors(false));
} catch (IOException e) {
  LOG.error("Bulk item errors: {}", e.getMessage());
}

When it happens

Trigger: Calling Beam's ElasticsearchIO.write() with .setThrowWriteErrors(true) while some documents in a bulk request fail (e.g. 400 mapping errors, document_parse_exception) — the bulk endpoint returns per-item errors even though the overall HTTP status is 200.

Common situations: Documents whose fields don't match the index mapping; index templates rejecting dynamic fields; ES nodes returning partial failures under load; type mismatch between doc and existing mapping after version upgrades.

Related errors

• Error writing to ES after %d attempt(s). No more attempts al
• Cannot get Elasticsearch version
• Illegal access to pipeline after visitor traversal was compl
• Pipeline update will not be possible because the following t
• Unrecognized value for stable unique names:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7f2efdb46f44ce3a. Report an issue: GitHub.