apache/beam · error · IllegalArgumentException

Batch size is too large! It should be smaller or equal than

Error message

Batch size is too large! It should be smaller or equal than %d.

What it means

DLPInspectText batches input elements and sends them to the Cloud DLP API, which enforces a maximum payload size (DLP_PAYLOAD_LIMIT_BYTES). build() throws this IllegalArgumentException when the configured batchSizeBytes exceeds that limit, since the resulting requests would be rejected by the service.

Source

Thrown at sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPInspectText.java:145

    public abstract Builder setColumnDelimiter(String delimiter);

    /**
     * Sets list of column names if the input KV value is a delimited row.
     *
     * @param headerColumns List of column names if the input KV value is a delimited row.
     */
    public abstract Builder setHeaderColumns(PCollectionView<List<String>> headerColumns);

    abstract DLPInspectText autoBuild();

    public DLPInspectText build() {
      DLPInspectText inspectText = autoBuild();
      if (inspectText.getInspectTemplateName() == null && inspectText.getInspectConfig() == null) {
        throw new IllegalArgumentException(
            "Either inspectTemplateName or inspectConfig must be supplied!");
      }
      if (inspectText.getBatchSizeBytes() > DLP_PAYLOAD_LIMIT_BYTES) {
        throw new IllegalArgumentException(
            String.format(
                "Batch size is too large! It should be smaller or equal than %d.",
                DLP_PAYLOAD_LIMIT_BYTES));
      }
      if (inspectText.getColumnDelimiter() == null && inspectText.getHeaderColumns() != null) {
        throw new IllegalArgumentException(
            "Column delimiter should be set if headers are present.");
      }
      if (inspectText.getHeaderColumns() == null && inspectText.getColumnDelimiter() != null) {
        throw new IllegalArgumentException(
            "Column headers should be supplied when delimiter is present.");
      }
      return inspectText;
    }
  }

  public static Builder newBuilder() {
    return new AutoValue_DLPInspectText.Builder();

View on GitHub (pinned to 12126d8942)

Solutions

Lower .setBatchSizeBytes(n) to a value less than or equal to DLP_PAYLOAD_LIMIT_BYTES.
Use the default batch size by omitting setBatchSizeBytes entirely.
If larger batches are needed for throughput, split the work across multiple transforms or reduce per-element size instead of raising the batch size.

Example fix

// before
.setBatchSizeBytes(20 * 1024 * 1024) // over the DLP payload limit
// after
.setBatchSizeBytes(5 * 1024 * 1024) // within DLP_PAYLOAD_LIMIT_BYTES

Defensive patterns

Strategy: validation

Validate before calling

long limit = 524288; // DLP_PAYLOAD_LIMIT_BYTES constant in DLPInspectText
if (batchSizeBytes > limit) {
  batchSizeBytes = limit;
}

Try / catch

try {
  DLPInspectText inspect = DLPInspectText.newBuilder()...setBatchSizeBytes(size).build();
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Batch size is too large")) {
    // clamp size to DLP_PAYLOAD_LIMIT_BYTES and rebuild
  }
}

Prevention

Reference DLP_PAYLOAD_LIMIT_BYTES instead of hardcoding a batch size.
Keep the default batch size unless profiling shows a need to change it.
Add a config sanity check that clamps batch size to the documented limit.

When it happens

Trigger: Calling DLPInspectText.newBuilder()...setBatchSizeBytes(n).build() where n is greater than DLP_PAYLOAD_LIMIT_BYTES (the DLP content API payload cap).

Common situations: Developers raise the batch size to improve throughput without checking the DLP API's content limit, or reuse a batch size tuned for a different API/service.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0d742cce467e0d53. Report an issue: GitHub.