apache/beam · error · IllegalArgumentException

Batch size is too large! It should be smaller or equal than

Error message

Batch size is too large! It should be smaller or equal than %d.

What it means

DLPReidentifyText.Builder.build() validates the configured byte batch size against DLP_PAYLOAD_LIMIT_BYTES, the maximum payload the Google DLP reidentify API accepts per request. If the user sets a batchSizeBytes larger than this limit, build() refuses to construct the transform. This fails fast at pipeline construction time so an oversized request is never sent to the DLP service.

Solutions

Lower batchSizeBytes to a value <= DLP_PAYLOAD_LIMIT_BYTES (or simply don't set it to use the default, which is already within the limit).
Split your input into smaller elements/PCollections so each DLP batch stays under the limit.
Consult the constant DLP_PAYLOAD_LIMIT_BYTES in DLPReidentifyText for the exact allowed maximum.

Example fix

// before
DLPReidentifyText.newBuilder().setBatchSizeBytes(2_000_000).build();
// after
DLPReidentifyText.newBuilder().setBatchSizeBytes(500_000).build(); // <= DLP_PAYLOAD_LIMIT_BYTES

Defensive patterns

Strategy: validation

Validate before calling

long payloadLimit = DLP_PAYLOAD_LIMIT_BYTES; // e.g. read from DLPReidentifyText
if (options.batchSizeBytes() > payloadLimit) {
  throw new IllegalArgumentException("batchSizeBytes must be <= " + payloadLimit);
}

Prevention

Don't set batchSizeBytes unless you have a reason; the default already respects the DLP limit.
Clamp configured batch size to DLP_PAYLOAD_LIMIT_BYTES when reading from config.
Unit-test builders with the max configured batch size.

When it happens

Trigger: Calling DLPReidentifyText.newBuilder().setBatchSizeBytes(n).build() where n > DLP_PAYLOAD_LIMIT_BYTES. The message's %d is filled with DLP_PAYLOAD_LIMIT_BYTES.

Common situations: Users assuming DLP can accept arbitrarily large text batches, copying a batch size from another DLP transform with a different limit, or setting batch size in a config knob that allows values above the API cap.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ae15185c62d5a44c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPReidentifyText.java:172

    /**
     * Sets ID of Google Cloud project to be used when deidentifying data.
     *
     * @param projectId ID of Google Cloud project to be used when deidentifying data.
     */
    public abstract Builder setProjectId(String projectId);

    abstract DLPReidentifyText autoBuild();

    public DLPReidentifyText build() {
      DLPReidentifyText dlpReidentifyText = autoBuild();
      if (dlpReidentifyText.getReidentifyConfig() == null
          && dlpReidentifyText.getReidentifyTemplateName() == null) {
        throw new IllegalArgumentException(
            "Either reidentifyConfig or reidentifyTemplateName need to be set!");
      }
      if (dlpReidentifyText.getBatchSizeBytes() > DLP_PAYLOAD_LIMIT_BYTES) {
        throw new IllegalArgumentException(
            String.format(
                "Batch size is too large! It should be smaller or equal than %d.",
                DLP_PAYLOAD_LIMIT_BYTES));
      }
      if (dlpReidentifyText.getColumnDelimiter() == null
          && dlpReidentifyText.getHeaderColumns() != null) {
        throw new IllegalArgumentException(
            "Column delimiter should be set if headers are present.");
      }
      if (dlpReidentifyText.getHeaderColumns() == null
          && dlpReidentifyText.getColumnDelimiter() != null) {
        throw new IllegalArgumentException(
            "Column headers should be supplied when delimiter is present.");
      }
      return dlpReidentifyText;
    }
  }

View on GitHub (pinned to 12126d8942)