apache/beam · error · IllegalArgumentException

Batch size is too large! It should be smaller or equal than

Error message

Batch size is too large! It should be smaller or equal than %d.

What it means

DLPReidentifyText.Builder.build() validates the configured byte batch size against DLP_PAYLOAD_LIMIT_BYTES, the maximum payload the Google DLP reidentify API accepts per request. If the user sets a batchSizeBytes larger than this limit, build() refuses to construct the transform. This fails fast at pipeline construction time so an oversized request is never sent to the DLP service.

Source

Thrown at sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPReidentifyText.java:172

    /**
     * Sets ID of Google Cloud project to be used when deidentifying data.
     *
     * @param projectId ID of Google Cloud project to be used when deidentifying data.
     */
    public abstract Builder setProjectId(String projectId);

    abstract DLPReidentifyText autoBuild();

    public DLPReidentifyText build() {
      DLPReidentifyText dlpReidentifyText = autoBuild();
      if (dlpReidentifyText.getReidentifyConfig() == null
          && dlpReidentifyText.getReidentifyTemplateName() == null) {
        throw new IllegalArgumentException(
            "Either reidentifyConfig or reidentifyTemplateName need to be set!");
      }
      if (dlpReidentifyText.getBatchSizeBytes() > DLP_PAYLOAD_LIMIT_BYTES) {
        throw new IllegalArgumentException(
            String.format(
                "Batch size is too large! It should be smaller or equal than %d.",
                DLP_PAYLOAD_LIMIT_BYTES));
      }
      if (dlpReidentifyText.getColumnDelimiter() == null
          && dlpReidentifyText.getHeaderColumns() != null) {
        throw new IllegalArgumentException(
            "Column delimiter should be set if headers are present.");
      }
      if (dlpReidentifyText.getHeaderColumns() == null
          && dlpReidentifyText.getColumnDelimiter() != null) {
        throw new IllegalArgumentException(
            "Column headers should be supplied when delimiter is present.");
      }
      return dlpReidentifyText;
    }
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Lower batchSizeBytes to a value <= DLP_PAYLOAD_LIMIT_BYTES (or simply don't set it to use the default, which is already within the limit).
2. Split your input into smaller elements/PCollections so each DLP batch stays under the limit.
3. Consult the constant DLP_PAYLOAD_LIMIT_BYTES in DLPReidentifyText for the exact allowed maximum.

Example fix

// before
DLPReidentifyText.newBuilder().setBatchSizeBytes(2_000_000).build();
// after
DLPReidentifyText.newBuilder().setBatchSizeBytes(500_000).build(); // <= DLP_PAYLOAD_LIMIT_BYTES

long payloadLimit = DLP_PAYLOAD_LIMIT_BYTES; // e.g. read from DLPReidentifyText
if (options.batchSizeBytes() > payloadLimit) {
  throw new IllegalArgumentException("batchSizeBytes must be <= " + payloadLimit);
}

When it happens

Trigger: Calling DLPReidentifyText.newBuilder().setBatchSizeBytes(n).build() where n > DLP_PAYLOAD_LIMIT_BYTES. The message's %d is filled with DLP_PAYLOAD_LIMIT_BYTES.

Common situations: Users assuming DLP can accept arbitrarily large text batches, copying a batch size from another DLP transform with a different limit, or setting batch size in a config knob that allows values above the API cap.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column delimiter should be set if headers are present.
• Column headers should be supplied when delimiter is present.
• Either startVersion or startTimestamp must be set.
• No filesystem found for scheme
• Missing required value for group [<group>]. At least one of

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ae15185c62d5a44c. Report an issue: GitHub.