apache/beam · error · ValueError

CombineFn spec missing type

Error message

CombineFn spec missing type: {fn_spec}

What it means

create_combine_fn inside Combine.expand raises ValueError when an aggregation's fn spec mapping has no 'type' key. Each combine fn spec must name either a builtin combine function or a Python callable source.

Solutions

1. Add fn: <name> (shorthand) or fn: {type: <name>} to each combine entry.
2. Use a builtin name (sum, count, mean, min, max, etc.) or supply a Python callable via source.
3. Validate each entry has both a destination field and a fn/type.

Example fix

// before
combine:
  total: {value: amount}
// after
combine:
  total: {fn: sum, value: amount}

def check_agg_specs(combine_cfg: dict):
    for dest, agg in combine_cfg.get('combine', {}).items():
        fn = agg.get('fn')
        if isinstance(fn, dict) and 'type' not in fn:
            raise SystemExit(f'combine.{dest}: fn spec missing type')
        if fn is None and 'fn' not in agg:
            raise SystemExit(f'combine.{dest}: missing fn')

def has_fn_type(agg: dict) -> bool:
    fn = agg.get('fn')
    return isinstance(fn, str) or (isinstance(fn, dict) and isinstance(fn.get('type'), str))

try:
    expand(pcoll)
except ValueError as e:
    if 'CombineFn spec missing type' in str(e):
        raise SystemExit('Add fn: <name> or fn: {type: <name>} to each combine entry') from e
    raise

When it happens

Trigger: YAML combine config entry like {value: amount, output: total} without fn/type, or a fn mapping like fn: {config: ...} missing type.

Common situations: Omitting 'fn'/'type' while only specifying 'value'; malformed copy-paste where the fn block was truncated.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Ambiguous expression type (perhaps missing quoting?)
• cross_product must be specified true or false when…
• Missing combine parameter in Combine config.
• Unknown CombineFn
• Unknown language for mapping transform

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a562de1152b0bbbe. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_combine.py:122

  def __init__(
      self,
      group_by: Iterable[str],
      combine: Mapping[str, Mapping[str, Any]],
      language: Optional[str] = None):
    self._group_by = group_by
    self._combine = combine
    self._language = language

  def expand(self, pcoll):
    input_types = dict(named_fields_from_element_type(pcoll.element_type))
    all_fields = list(input_types.keys())
    unknown_keys = set(self._group_by) - set(all_fields)
    if unknown_keys:
      raise ValueError(f'Unknown grouping columns: {list(unknown_keys)}')

    def create_combine_fn(fn_spec):
      if 'type' not in fn_spec:
        raise ValueError(f'CombineFn spec missing type: {fn_spec}')
      elif fn_spec['type'] in BUILTIN_COMBINE_FNS:
        return BUILTIN_COMBINE_FNS[fn_spec['type']]
      elif self._language == 'python':
        # TODO(yaml): Support output_type here as well.
        fn = python_callable.PythonCallableWithSource.load_from_source(
            fn_spec['type'])
        if 'config' in fn_spec:
          fn = fn(**fn_spec['config'])
        return fn
      else:
        raise TypeError('Unknown CombineFn: {fn_spec}')

    def extract_return_type(expr):
      if isinstance(expr, str) and expr in input_types:
        return input_types[expr]
      expr_hints = get_type_hints(expr)
      if (expr_hints and expr_hints.has_simple_output_type() and
          expr_hints.simple_output_type(None) != typehints.Any):

View on GitHub (pinned to 12126d8942)