{"record":{"id":"157cd0e1abb15db9","repo":"apache/beam","slug":"error-writing-to-es-after-d-attempt-s-no-more-attempts","errorCode":null,"errorMessage":"Error writing to ES after %d attempt(s). No more attempts allowed","messagePattern":"Error writing to ES after (.+?) attempt\\(s\\)\\. No more attempts allowed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java","lineNumber":2984,"sourceCode":"            request.setEntity(requestBody);\n            response = restClient.performRequest(request);\n            responseEntity = new BufferedHttpEntity(response.getEntity());\n          } catch (java.io.IOException ex) {\n            if (isRetryableClientException(ex)) {\n              LOG.error(\"Caught ES timeout, retrying\", ex);\n              continue;\n            }\n          }\n          // if response has no 429 errors\n          if (!Objects.requireNonNull(spec.getRetryConfiguration())\n              .getRetryPredicate()\n              .test(responseEntity)) {\n            return responseEntity;\n          } else {\n            LOG.warn(\"ES Cluster is responding with HTP 429 - TOO_MANY_REQUESTS.\");\n          }\n        }\n        throw new IOException(String.format(RETRY_FAILED_LOG, attempt));\n      }\n\n      @Teardown\n      public void closeClient() throws IOException {\n        if (restClient != null) {\n          restClient.close();\n        }\n      }\n    }\n  }\n\n  private static void maybeLogVersionDeprecationWarning(int clusterVersion) {\n    if (DEPRECATED_CLUSTER_VERSIONS.contains(clusterVersion)) {\n      LOG.warn(\n          \"Support for Elasticsearch cluster version {} will be dropped in a future release of \"\n              + \"the Apache Beam SDK\",\n          clusterVersion);\n    }","sourceCodeStart":2966,"sourceCodeEnd":3002,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L2966-L3002","documentation":"WriteToElasticsearch retries bulk writes that fail with HTTP 429 (TOO_MANY_REQUESTS) up to a configured number of attempts. When the retry budget is exhausted without receiving an accepted response, it throws IOException with RETRY_FAILED_LOG indicating the attempt count. This means Elasticsearch is persistently rejecting or failing writes under load.","triggerScenarios":"ElasticsearchIO.write() with .setUseCreated(false)/retry configuration hitting repeated HTTP 429 responses from the bulk API across all attempts (setNumRetries exhausted, setRetryPause too short).","commonSituations":"Bulk index overload on undersized ES clusters; write thread pool saturation; disk watermark exceeded causing 429s; beam workers sending too large batches in parallel.","solutions":["Increase .setNumRetries(...) and .setRetryPause(...) on the Write transform","Reduce batch size (.withMaxBatchSize / .withMaxBatchByteSize) and pipeline parallelism to lower write pressure","Scale up the Elasticsearch cluster or increase the thread pool / shard count","Check ES for disk watermarks (flood_stage) blocking writes and free disk space","Prefer try-catch is not possible in DoFn — let Beam retry the bundle via RunnerOverrides"],"exampleFix":"// before\nElasticsearchIO.Write.with(name, connection);\n// after\nElasticsearchIO.Write.with(name, connection)\n    .setNumRetries(10)\n    .setRetryPause(Duration.millis(5000))\n    .withMaxBatchSize(500);","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Configure generous retries upfront; retry is handled internally per attempt\nElasticsearchIO.write()\n  .setNumRetries(10)\n  .setRetryPause(org.joda.time.Duration.millis(5000));","preventionTips":["Size batches (withMaxBatchSize/withMaxBatchByteSize) to keep bulk responses under ES thread-pool limits","Monitor ES thread_pool.write.queue and disk watermarks","Add backpressure by limiting Beam worker parallelism for ES-heavy pipelines"],"tags":["java","elasticsearch","beam","retry","rate-limit"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}