apache/beam · error · ValueError

target_batch_overhead

Error message

target_batch_overhead (%s) must be between 0 and 1

What it means

Raised in _BatchSizeEstimator (BatchElements) __init__ when target_batch_overhead is truthy but outside the (0, 1] interval. The overhead is a fraction of per-element cost to target (e.g. 0.05 for 5% overhead), so values like 0, negative numbers, or >1 are meaningless and rejected.

Solutions

  1. Express the overhead as a fraction in (0, 1], e.g. 0.02 for 2%.
  2. If your config stores percentages, divide by 100 before passing: target_batch_overhead=overhead_pct / 100.0.
  3. Validate the value at the call site and reject/normalize anything outside 0 < x <= 1 before constructing the transform.

Example fix

// before
beam.BatchElements(target_batch_overhead=25)  # percent, not fraction

// after
beam.BatchElements(target_batch_overhead=0.25)
Defensive patterns

Strategy: validation

Validate before calling

if target_batch_overhead is not None and not 0 < target_batch_overhead <= 1:
    raise ValueError('target_batch_overhead must be a fraction in (0, 1], e.g. 0.05 for 5%')

Type guard

def valid_overhead(x):
    return x is None or (isinstance(x, (int, float)) and 0 < x <= 1)

Try / catch

try:
    t = beam.BatchElements(target_batch_overhead=overhead)
except ValueError:
    logging.warning('target_batch_overhead=%s invalid; defaulting to 0.05', overhead)
    t = beam.BatchElements(target_batch_overhead=0.05)

Prevention

When it happens

Trigger: BatchElements(target_batch_overhead=1.5), =-0.1, or =0 (0 also fails because the condition is 0 < x <= 1 when truthy).

Common situations: Confusing a fraction with a percentage (passing 50 instead of 0.05); reading the value from config as 'percent' and forgetting to divide by 100; typos like a doubled decimal.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3eb71e41ed3a7d65. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/util.py:525

  _MAX_GROWTH_FACTOR = 2

  def __init__(
      self,
      min_batch_size=1,
      max_batch_size=10000,
      target_batch_overhead=.05,
      target_batch_duration_secs=10,
      target_batch_duration_secs_including_fixed_cost=None,
      variance=0.25,
      clock=time.time,
      ignore_first_n_seen_per_batch_size=0,
      record_metrics=True):
    if min_batch_size > max_batch_size:
      raise ValueError(
          "Minimum (%s) must not be greater than maximum (%s)" %
          (min_batch_size, max_batch_size))
    if target_batch_overhead and not 0 < target_batch_overhead <= 1:
      raise ValueError(
          "target_batch_overhead (%s) must be between 0 and 1" %
          (target_batch_overhead))
    if target_batch_duration_secs and target_batch_duration_secs <= 0:
      raise ValueError(
          "target_batch_duration_secs (%s) must be positive" %
          (target_batch_duration_secs))
    if (target_batch_duration_secs_including_fixed_cost and
        target_batch_duration_secs_including_fixed_cost <= 0):
      raise ValueError(
          "target_batch_duration_secs_including_fixed_cost "
          "(%s) must be positive" %
          (target_batch_duration_secs_including_fixed_cost))
    if not (target_batch_overhead or target_batch_duration_secs or
            target_batch_duration_secs_including_fixed_cost):
      raise ValueError(
          "At least one of target_batch_overhead or "
          "target_batch_duration_secs or "
          "target_batch_duration_secs_including_fixed_cost must be positive.")

View on GitHub (pinned to 12126d8942)