Use of withMaxBufferingDuration requires…

Error message

Use of withMaxBufferingDuration requires withUseStatefulBatches(true). Setting that automatically.

What it means

ElasticsearchIO's bulk batching with a max buffering duration is implemented on top of Beam's stateful processing, which requires stateful batches to be enabled. Calling withMaxBufferingDuration alone would produce a silently ignored setting, so the sink logs this warning and automatically enables withUseStatefulBatches(true) while applying the requested duration.

Solutions

Explicitly call .withUseStatefulBatches(true) before .withMaxBufferingDuration(...) to make the dependency clear
Accept the automatic enabling — no action strictly required, but be aware stateful batches change pipeline execution
Review whether a max buffering duration is needed at all; the default batch-size-driven flushing may suffice

Example fix

// before
ElasticsearchIO.write().withConnectionConfiguration(cfg).withMaxBufferingDuration(Duration.standardSeconds(5))
// after
ElasticsearchIO.write().withConnectionConfiguration(cfg).withUseStatefulBatches(true).withMaxBufferingDuration(Duration.standardSeconds(5))

Defensive patterns

Strategy: validation

Validate before calling

if (maxBufferingDuration != null && !write.withUseStatefulBatches) { write = write.withUseStatefulBatches(true); } // ensure stateful batches precede withMaxBufferingDuration(...)

Prevention

Always chain withUseStatefulBatches(true) directly before withMaxBufferingDuration(...)
Centralize the ElasticsearchIO sink builder in a helper so buffering options are set consistently
Watch startup logs for automatic-setting warnings after pipeline upgrades

When it happens

Trigger: Calling BulkIO.withMaxBufferingDuration(Duration) on an ElasticsearchIO write without having previously called withUseStatefulBatches(true).

Common situations: Developers tuning Elasticsearch bulk flush latency set a buffering timeout to avoid waiting for maxBatchSize to fill, forgetting that the timeout only works in stateful-batch mode.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/14cea8a44a43613b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java:2522

        allowableResponseErrorTypes = new HashSet<>();
      }

      return builder().setAllowedResponseErrors(allowableResponseErrorTypes).build();
    }

    /**
     * If using {@link BulkIO#withUseStatefulBatches}, this can be used to set a maximum elapsed
     * time before buffered elements are emitted to Elasticsearch as a Bulk API request. If this
     * config is not set, Bulk requests will not be issued until {@link BulkIO#getMaxBatchSize}
     * number of documents have been buffered. This may result in higher latency in particular if
     * your max batch size is set to a large value and your pipeline input is low volume.
     *
     * @param maxBufferingDuration the maximum duration to wait before sending any buffered
     *     documents to Elasticsearch, regardless of maxBatchSize.
     * @return the {@link BulkIO} with maximum buffering duration set
     */
    public BulkIO withMaxBufferingDuration(Duration maxBufferingDuration) {
      LOG.warn(
          "Use of withMaxBufferingDuration requires withUseStatefulBatches(true). "
              + "Setting that automatically.");
      return builder()
          .setUseStatefulBatches(true)
          .setMaxBufferingDuration(maxBufferingDuration)
          .build();
    }

    /**
     * Whether or not to use Stateful Processing to ensure bulk requests have the desired number of
     * entities i.e. as close to the maxBatchSize as possible. By default without this feature
     * enabled, Bulk requests will not contain more than maxBatchSize entities, but the lower bound
     * of batch size is determined by Beam Runner bundle sizes, which may be as few as 1.
     *
     * @param useStatefulBatches true enables the use of Stateful Processing to ensure that batches
     *     are as close to the maxBatchSize as possible.
     * @return the {@link BulkIO} with Stateful Processing enabled or disabled
     */

View on GitHub (pinned to 12126d8942)