apache/beam · error · IllegalArgumentException

smallBatchTag and largeBatchTag must be different

Error message

smallBatchTag and largeBatchTag must be different

What it means

BundleLifter's constructor validates that the two state tags used to distinguish small and large batches refer to different state IDs. If both tags share the same ID, small and large batch state would collide in the batched state map, so the library refuses to construct the lifter.

Source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/BundleLifter.java:136

        taggedOutput.output(element);
      }
    }
  }

  private BundleLifter(TupleTag<T> smallBatchTag, TupleTag<T> largeBatchTag, int threshold) {
    this(smallBatchTag, largeBatchTag, threshold, x -> 1);
  }

  private BundleLifter(
      TupleTag<T> smallBatchTag,
      TupleTag<T> largeBatchTag,
      int threshold,
      SerializableFunction<T, Integer> elementSizer) {
    if (smallBatchTag == null || largeBatchTag == null) {
      throw new IllegalArgumentException("smallBatchTag and largeBatchTag must not be null");
    }
    if (smallBatchTag.getId().equals(largeBatchTag.getId())) {
      throw new IllegalArgumentException("smallBatchTag and largeBatchTag must be different");
    }
    if (threshold <= 0) {
      throw new IllegalArgumentException("Threshold must be a positive integer");
    }

    this.smallBatchTag = smallBatchTag;
    this.largeBatchTag = largeBatchTag;
    this.threshold = threshold;
    this.elementSizer = elementSizer;
  }

  public static <T> BundleLifter<T> of(
      TupleTag<T> smallBatchTag, TupleTag<T> largeBatchTag, int threshold) {
    return new BundleLifter<>(smallBatchTag, largeBatchTag, threshold);
  }

  public static <T> BundleLifter<T> of(
      TupleTag<T> smallBatchTag,

View on GitHub (pinned to 12126d8942)

Solutions

Create two distinct StateTag instances with different identities/parameters for small and large batch tags.
If the tags are built programmatically, make their getId() inputs differ (e.g., different names or components).
Log both tag IDs before constructing BundleLifter to verify they differ.

Example fix

// before
StateTag<Object, List<T>> batchTag = StateTags.key("batch");
BundleLifter<T> lifter = new BundleLifter<>(td, cls, supplier, batchTag, batchTag, 100, sizer);
// after
StateTag<Object, List<T>> smallBatchTag = StateTags.key("small-batch");
StateTag<Object, List<T>> largeBatchTag = StateTags.key("large-batch");
BundleLifter<T> lifter = new BundleLifter<>(td, cls, supplier, smallBatchTag, largeBatchTag, 100, sizer);

Defensive patterns

Strategy: validation

Validate before calling

if (smallBatchTag == null || largeBatchTag == null) throw new IllegalArgumentException("tags must not be null");
checkState(!smallBatchTag.getId().equals(largeBatchTag.getId()), "smallBatchTag and largeBatchTag must differ");

Prevention

Define small and large batch tags as separate static final constants with clearly different names.
Never reuse a single StateTag variable for both constructor parameters.

When it happens

Trigger: Calling new BundleLifter<>(typeDescriptor, batcherClass, batcherSupplier, smallBatchTag, largeBatchTag, threshold, elementSizer) where smallBatchTag.getId().equals(largeBatchTag.getId()) — typically because the same StateTag (or two tags built with identical parameters, since getId() derives from them) was passed for both parameters.

Common situations: Copy-pasting a single StateTag definition and reusing it for both parameters; constructing two tags from the same template with identical inputs and assuming they are distinct; refactoring that accidentally collapsed two tag constants into one.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ad12f31ec08d878c. Report an issue: GitHub.