apache/beam · error · IllegalArgumentException

Batch size is too large! It should be smaller or equal than

Error message

Batch size is too large! It should be smaller or equal than %d.

What it means

DLPDeidentifyText.Builder.build enforces the DLP payload limit (DLP_PAYLOAD_LIMIT_BYTES). If the configured batchSizeBytes exceeds it, build throws IllegalArgumentException because the DLP API would reject the request as too large.

Solutions

1. Lower batchSizeBytes to at or below DLP_PAYLOAD_LIMIT_BYTES (setBatchSizeBytes(DLP_PAYLOAD_LIMIT_BYTES)).
2. Compute batch sizes from average record size to stay under the limit.
3. Remove the explicit size and rely on the default safe batch size.

Example fix

// before
.setBatchSizeBytes(50_000_000)
// after
.setBatchSizeBytes(DLP_PAYLOAD_LIMIT_BYTES) // e.g. within the allowed max

if (batchSizeBytes > DLP_PAYLOAD_LIMIT_BYTES) batchSizeBytes = DLP_PAYLOAD_LIMIT_BYTES;

When it happens

Trigger: Calling .setBatchSizeBytes(n) with n greater than DLP_PAYLOAD_LIMIT_BYTES and then building the transform.

Common situations: Setting a very large batch to reduce API calls; misremembering the DLP content limit; converting record counts to bytes incorrectly.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Either deidentifyConfig or deidentifyTemplateName need to…
• Batch size is too large! It should be smaller or equal than
• Batch size is too large! It should be smaller or equal than
• Column delimiter should be set if headers are present.
• Column delimiter should be set if headers are present.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b4e00f55622cf045. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPDeidentifyText.java:167

     * Sets configuration object for data deidentification. If present, supersedes the template
     * settings.
     *
     * @param deidentifyConfig Configuration object for data deidentification. If present,
     *     supersedes the template settings.
     */
    public abstract Builder setDeidentifyConfig(DeidentifyConfig deidentifyConfig);

    abstract DLPDeidentifyText autoBuild();

    public DLPDeidentifyText build() {
      DLPDeidentifyText dlpDeidentifyText = autoBuild();
      if (dlpDeidentifyText.getDeidentifyConfig() == null
          && dlpDeidentifyText.getDeidentifyTemplateName() == null) {
        throw new IllegalArgumentException(
            "Either deidentifyConfig or deidentifyTemplateName need to be set!");
      }
      if (dlpDeidentifyText.getBatchSizeBytes() > DLP_PAYLOAD_LIMIT_BYTES) {
        throw new IllegalArgumentException(
            String.format(
                "Batch size is too large! It should be smaller or equal than %d.",
                DLP_PAYLOAD_LIMIT_BYTES));
      }
      if (dlpDeidentifyText.getColumnDelimiter() == null
          && dlpDeidentifyText.getHeaderColumns() != null) {
        throw new IllegalArgumentException(
            "Column delimiter should be set if headers are present.");
      }
      if (dlpDeidentifyText.getHeaderColumns() == null
          && dlpDeidentifyText.getColumnDelimiter() != null) {
        throw new IllegalArgumentException(
            "Column headers should be supplied when delimiter is present.");
      }
      return dlpDeidentifyText;
    }
  }

View on GitHub (pinned to 12126d8942)