{"record":{"id":"9b35ac02c59b6a77","repo":"apache/beam","slug":"unknown-grouping-columns-list-unknown-keys","errorCode":null,"errorMessage":"Unknown grouping columns: {list(unknown_keys)}","messagePattern":"Unknown grouping columns: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_combine.py","lineNumber":118,"sourceCode":"    combine: The aggregation function to use.\n    language: The language used to define (and execute) the\n      custom callables in `combine`. Defaults to generic.\n  \"\"\"\n  def __init__(\n      self,\n      group_by: Iterable[str],\n      combine: Mapping[str, Mapping[str, Any]],\n      language: Optional[str] = None):\n    self._group_by = group_by\n    self._combine = combine\n    self._language = language\n\n  def expand(self, pcoll):\n    input_types = dict(named_fields_from_element_type(pcoll.element_type))\n    all_fields = list(input_types.keys())\n    unknown_keys = set(self._group_by) - set(all_fields)\n    if unknown_keys:\n      raise ValueError(f'Unknown grouping columns: {list(unknown_keys)}')\n\n    def create_combine_fn(fn_spec):\n      if 'type' not in fn_spec:\n        raise ValueError(f'CombineFn spec missing type: {fn_spec}')\n      elif fn_spec['type'] in BUILTIN_COMBINE_FNS:\n        return BUILTIN_COMBINE_FNS[fn_spec['type']]\n      elif self._language == 'python':\n        # TODO(yaml): Support output_type here as well.\n        fn = python_callable.PythonCallableWithSource.load_from_source(\n            fn_spec['type'])\n        if 'config' in fn_spec:\n          fn = fn(**fn_spec['config'])\n        return fn\n      else:\n        raise TypeError('Unknown CombineFn: {fn_spec}')\n\n    def extract_return_type(expr):\n      if isinstance(expr, str) and expr in input_types:","sourceCodeStart":100,"sourceCodeEnd":136,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_combine.py#L100-L136","documentation":"Expand of the Combine transform validates every column in group_by exists in the input PCollection's schema. Any grouping key not present in the input element type raises ValueError listing the unknown columns.","triggerScenarios":"YAML Combine config with group_by referencing fields absent from the input schema (typo, wrong case, nested field given as a plain name, or upstream transform changed the schema).","commonSituations":"Schema drift after editing an upstream Read or Parse; case mismatch between YAML and Avro/JSON field names; grouping on an output field created later in the pipeline.","solutions":["Print the input schema (e.g. with --output_json or a LogForTesting transform) and correct group_by names to match exactly.","Fix typos/casing so each group_by entry matches an existing input field.","Reorder the pipeline so grouping happens after the transform that produces the referenced fields.","Use nested field access syntax (e.g. 'a.b') only if the schema actually has nested rows."],"exampleFix":"// before\nconfig:\n  group_by: [userId]\n  combine: {total: {fn: sum, value: amount}}\n// after\nconfig:\n  group_by: [user_id]\n  combine: {total: {fn: sum, value: amount}}","handlingStrategy":"validation","validationCode":"def check_group_by(schema_fields: list, group_by: list):\n    unknown = set(group_by) - set(schema_fields)\n    if unknown:\n        raise SystemExit(f'group_by fields not in input schema: {sorted(unknown)}; available: {schema_fields}')","typeGuard":"def group_by_is_valid(element_type, group_by: list) -> bool:\n    from apache_beam.typehints.schemas import named_fields_from_element_type\n    fields = {name for name, _ in named_fields_from_element_type(element_type)}\n    return set(group_by) <= fields","tryCatchPattern":"try:\n    result = combine_transform.expand(pcoll)\nexcept ValueError as e:\n    if 'Unknown grouping columns' in str(e):\n        log_input_schema(pcoll); raise SystemExit('Fix group_by names to match input schema') from e\n    raise","preventionTips":["Print the input schema before authoring group_by lists","Match field names exactly, including case","Re-check group_by after any upstream schema change","Use CI schema tests that validate YAML references against input schemas"],"tags":["yaml","schema","beam-yaml"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}