apache/beam · error · ValueError

max_batch_size ({max_batch_size}) must be >= min_batch_size

Error message

max_batch_size ({max_batch_size}) must be >= min_batch_size ({min_batch_size})

What it means

GroupIntoBatches parameters require max_batch_size >= min_batch_size; the constructor raises ValueError naming both values when the ordering is violated. It fires immediately at transform construction time.

Source

Thrown at sdks/python/apache_beam/transforms/util.py:1388

      # Batch with custom size function
      data = [{'text': 'short'}, {'text': 'medium text'},
              {'text': 'long text here'}]
      batched = data | SortAndBatchElements(
          min_batch_size=1,
          max_batch_size=10,
          max_batch_weight=100,
          element_size_fn=lambda x: len(x['text']))
  """
  def __init__(
      self,
      min_batch_size: int,
      max_batch_size: int,
      max_batch_weight: int,
      element_size_fn: Optional[Callable[[Any], int]] = None):
    if min_batch_size < 1:
      raise ValueError(f'min_batch_size must be >= 1, got {min_batch_size}')
    if max_batch_size < min_batch_size:
      raise ValueError(
          f'max_batch_size ({max_batch_size}) must be >= '
          f'min_batch_size ({min_batch_size})')
    if max_batch_weight < 1:
      raise ValueError(f'max_batch_weight must be >= 1, got {max_batch_weight}')
    if element_size_fn is not None and not callable(element_size_fn):
      raise TypeError('element_size_fn must be callable')

    self._min_batch_size = min_batch_size
    self._max_batch_size = max_batch_size
    self._max_batch_weight = max_batch_weight

    # None means the DoFn will use its own _default_element_size method,
    # which tries len() and warns once on TypeError before falling back to 1.
    self._element_size_fn = element_size_fn

  def expand(self, pcoll):
    if pcoll.windowing.is_default():
      return pcoll | ParDo(

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure max_batch_size >= min_batch_size at the call site
2. Clamp programmatically: max_batch_size = max(max_batch_size, min_batch_size)
3. Add cross-field validation where the two values are read from config

Example fix

// before
util.GroupIntoBatches(min_batch_size=50, max_batch_size=10, max_batch_weight=1000)
// after
min_bs = 50
max_bs = max(10, min_bs)
util.GroupIntoBatches(min_batch_size=min_bs, max_batch_size=max_bs, max_batch_weight=1000)

if max_batch_size < min_batch_size:
    raise ValueError('max_batch_size must be >= min_batch_size')

When it happens

Trigger: Constructing util.GroupIntoBatches with max_batch_size smaller than min_batch_size, e.g. GroupIntoBatches(min_batch_size=50, max_batch_size=10, ...) or values swapped when parameterized from config.

Common situations: Swapping arguments in positional/keyword order; independent config knobs that are validated separately and never cross-checked; tuning scripts shrinking max_batch_size below the fixed min.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• min_batch_size must be >= 1, got {min_batch_size}
• max_batch_weight must be >= 1, got {max_batch_weight}
• element_size_fn must be callable
• If `num_buckets` is set, it has to be an integer greater tha
• The size parameter must be strictly positive.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a01dff1c37852a0f. Report an issue: GitHub.