apache/beam · error · ValueError

Unknown grouping columns

Error message

Unknown grouping columns: {list(unknown_keys)}

What it means

Expand of the Combine transform validates every column in group_by exists in the input PCollection's schema. Any grouping key not present in the input element type raises ValueError listing the unknown columns.

Solutions

1. Print the input schema (e.g. with --output_json or a LogForTesting transform) and correct group_by names to match exactly.
2. Fix typos/casing so each group_by entry matches an existing input field.
3. Reorder the pipeline so grouping happens after the transform that produces the referenced fields.
4. Use nested field access syntax (e.g. 'a.b') only if the schema actually has nested rows.

Example fix

// before
config:
  group_by: [userId]
  combine: {total: {fn: sum, value: amount}}
// after
config:
  group_by: [user_id]
  combine: {total: {fn: sum, value: amount}}

def check_group_by(schema_fields: list, group_by: list):
    unknown = set(group_by) - set(schema_fields)
    if unknown:
        raise SystemExit(f'group_by fields not in input schema: {sorted(unknown)}; available: {schema_fields}')

def group_by_is_valid(element_type, group_by: list) -> bool:
    from apache_beam.typehints.schemas import named_fields_from_element_type
    fields = {name for name, _ in named_fields_from_element_type(element_type)}
    return set(group_by) <= fields

try:
    result = combine_transform.expand(pcoll)
except ValueError as e:
    if 'Unknown grouping columns' in str(e):
        log_input_schema(pcoll); raise SystemExit('Fix group_by names to match input schema') from e
    raise

When it happens

Trigger: YAML Combine config with group_by referencing fields absent from the input schema (typo, wrong case, nested field given as a plain name, or upstream transform changed the schema).

Common situations: Schema drift after editing an upstream Read or Parse; case mismatch between YAML and Avro/JSON field names; grouping on an output field created later in the pipeline.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Can only use expressions on a schema'd input.
• violates schema
• The input to this transform does not appear to be an error…
• Unknown or unsupported atomic type
• Unrecognized type_info

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9b35ac02c59b6a77. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_combine.py:118

    combine: The aggregation function to use.
    language: The language used to define (and execute) the
      custom callables in `combine`. Defaults to generic.
  """
  def __init__(
      self,
      group_by: Iterable[str],
      combine: Mapping[str, Mapping[str, Any]],
      language: Optional[str] = None):
    self._group_by = group_by
    self._combine = combine
    self._language = language

  def expand(self, pcoll):
    input_types = dict(named_fields_from_element_type(pcoll.element_type))
    all_fields = list(input_types.keys())
    unknown_keys = set(self._group_by) - set(all_fields)
    if unknown_keys:
      raise ValueError(f'Unknown grouping columns: {list(unknown_keys)}')

    def create_combine_fn(fn_spec):
      if 'type' not in fn_spec:
        raise ValueError(f'CombineFn spec missing type: {fn_spec}')
      elif fn_spec['type'] in BUILTIN_COMBINE_FNS:
        return BUILTIN_COMBINE_FNS[fn_spec['type']]
      elif self._language == 'python':
        # TODO(yaml): Support output_type here as well.
        fn = python_callable.PythonCallableWithSource.load_from_source(
            fn_spec['type'])
        if 'config' in fn_spec:
          fn = fn(**fn_spec['config'])
        return fn
      else:
        raise TypeError('Unknown CombineFn: {fn_spec}')

    def extract_return_type(expr):
      if isinstance(expr, str) and expr in input_types:

View on GitHub (pinned to 12126d8942)