apache/beam · error · IOException

Error writing to ES after

Error message

Error writing to ES after %d attempt(s). No more attempts allowed

What it means

WriteToElasticsearch retries bulk writes that fail with HTTP 429 (TOO_MANY_REQUESTS) up to a configured number of attempts. When the retry budget is exhausted without receiving an accepted response, it throws IOException with RETRY_FAILED_LOG indicating the attempt count. This means Elasticsearch is persistently rejecting or failing writes under load.

Solutions

Increase .setNumRetries(...) and .setRetryPause(...) on the Write transform
Reduce batch size (.withMaxBatchSize / .withMaxBatchByteSize) and pipeline parallelism to lower write pressure
Scale up the Elasticsearch cluster or increase the thread pool / shard count
Check ES for disk watermarks (flood_stage) blocking writes and free disk space
Prefer try-catch is not possible in DoFn — let Beam retry the bundle via RunnerOverrides

Example fix

// before
ElasticsearchIO.Write.with(name, connection);
// after
ElasticsearchIO.Write.with(name, connection)
    .setNumRetries(10)
    .setRetryPause(Duration.millis(5000))
    .withMaxBatchSize(500);

Defensive patterns

Strategy: retry

Try / catch

// Configure generous retries upfront; retry is handled internally per attempt
ElasticsearchIO.write()
  .setNumRetries(10)
  .setRetryPause(org.joda.time.Duration.millis(5000));

Prevention

Size batches (withMaxBatchSize/withMaxBatchByteSize) to keep bulk responses under ES thread-pool limits
Monitor ES thread_pool.write.queue and disk watermarks
Add backpressure by limiting Beam worker parallelism for ES-heavy pipelines

When it happens

Trigger: ElasticsearchIO.write() with .setUseCreated(false)/retry configuration hitting repeated HTTP 429 responses from the bulk API across all attempts (setNumRetries exhausted, setRetryPause too short).

Common situations: Bulk index overload on undersized ES clusters; write thread pool saturation; disk watermark exceeded causing 429s; beam workers sending too large batches in parallel.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/157cd0e1abb15db9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java:2984

            request.setEntity(requestBody);
            response = restClient.performRequest(request);
            responseEntity = new BufferedHttpEntity(response.getEntity());
          } catch (java.io.IOException ex) {
            if (isRetryableClientException(ex)) {
              LOG.error("Caught ES timeout, retrying", ex);
              continue;
            }
          }
          // if response has no 429 errors
          if (!Objects.requireNonNull(spec.getRetryConfiguration())
              .getRetryPredicate()
              .test(responseEntity)) {
            return responseEntity;
          } else {
            LOG.warn("ES Cluster is responding with HTP 429 - TOO_MANY_REQUESTS.");
          }
        }
        throw new IOException(String.format(RETRY_FAILED_LOG, attempt));
      }

      @Teardown
      public void closeClient() throws IOException {
        if (restClient != null) {
          restClient.close();
        }
      }
    }
  }

  private static void maybeLogVersionDeprecationWarning(int clusterVersion) {
    if (DEPRECATED_CLUSTER_VERSIONS.contains(clusterVersion)) {
      LOG.warn(
          "Support for Elasticsearch cluster version {} will be dropped in a future release of "
              + "the Apache Beam SDK",
          clusterVersion);
    }

View on GitHub (pinned to 12126d8942)