apache/beam · error · TypeError

element_size_fn must be callable

Error message

element_size_fn must be callable

What it means

GroupIntoBatches' element_size_fn, when provided, must be callable; the constructor raises TypeError otherwise. The function computes each element's weight for batch accumulation, so a non-callable default (e.g. a constant int passed by mistake) is rejected.

Source

Thrown at sdks/python/apache_beam/transforms/util.py:1394

          max_batch_weight=100,
          element_size_fn=lambda x: len(x['text']))
  """
  def __init__(
      self,
      min_batch_size: int,
      max_batch_size: int,
      max_batch_weight: int,
      element_size_fn: Optional[Callable[[Any], int]] = None):
    if min_batch_size < 1:
      raise ValueError(f'min_batch_size must be >= 1, got {min_batch_size}')
    if max_batch_size < min_batch_size:
      raise ValueError(
          f'max_batch_size ({max_batch_size}) must be >= '
          f'min_batch_size ({min_batch_size})')
    if max_batch_weight < 1:
      raise ValueError(f'max_batch_weight must be >= 1, got {max_batch_weight}')
    if element_size_fn is not None and not callable(element_size_fn):
      raise TypeError('element_size_fn must be callable')

    self._min_batch_size = min_batch_size
    self._max_batch_size = max_batch_size
    self._max_batch_weight = max_batch_weight

    # None means the DoFn will use its own _default_element_size method,
    # which tries len() and warns once on TypeError before falling back to 1.
    self._element_size_fn = element_size_fn

  def expand(self, pcoll):
    if pcoll.windowing.is_default():
      return pcoll | ParDo(
          _SortAndBatchElementsDoFn(
              self._min_batch_size,
              self._max_batch_size,
              self._max_batch_weight,
              self._element_size_fn))
    return pcoll | ParDo(

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass an actual callable, e.g. element_size_fn=lambda x: len(str(x))
2. If you want a constant size, wrap it: element_size_fn=lambda x: 10
3. Print/type-check the argument before constructing the transform

Example fix

// before
util.GroupIntoBatches(1, 100, 1024, element_size_fn=16)
// after
util.GroupIntoBatches(1, 100, 1024, element_size_fn=lambda x: 16)

if element_size_fn is not None and not callable(element_size_fn):
    raise TypeError('element_size_fn must be callable')

def is_size_fn(v): return v is None or callable(v)

When it happens

Trigger: Passing element_size_fn=5 or a dict instead of a function, or passing a lambda-wannabe like `element_size_fn=lambda` syntax errors resolved to None-adjacent mistakes; also passing functools.partial results that turned out to be plain values.

Common situations: Confusing element_size_fn with a static element size (passing an int instead of len-like callable); accidental shadowing of a function name by a variable earlier in scope.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• source must be an UnboundedSource, got %r
• min_batch_size must be >= 1, got {min_batch_size}
• max_batch_size ({max_batch_size}) must be >= min_batch_size
• max_batch_weight must be >= 1, got {max_batch_weight}
• Unable to convert objects of type %s to a PCollection

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/11d40bbe1ff7d827. Report an issue: GitHub.