apache/beam · error · ValueError

Missing language specification, unknown input fields, or…

Error message

Missing language specification, unknown input fields, or invalid generic expression: {expr}. The given input fields are {input_fields}. See https://beam.apache.org/documentation/sdks/yaml-udf/#generic

What it means

After checking the map form, validate_generic_expression verifies a bare-string expression is either atomic (a known input field or literal) or a simple comparison of atomic operands. This final ValueError fires when the string expression references unknown input fields or is too complex to be handled generically, and no language was specified.

Solutions

1. Fix field names to match the input schema (compare against input_fields).
2. Simplify the expression to atomic terms or simple comparisons (==, <=, >=, <, >, !=).
3. Specify a language explicitly via the map form: {expression: ..., language: python}.
4. Consult the YAML UDF docs at the URL in the message for supported generic expressions.

Example fix

# before
expression: "toatal + amount"
# after
expression: "total + amount"  # or
configuration:
  language: python
  expression: "total + amount"

fields = set(input['schema'].keys())
used = re.findall(r'[A-Za-z_]\w*', expr)
unknown = set(used) - fields - {'True', 'False', 'None', 'and', 'or', 'not', 'in'}
assert not unknown, f'unknown fields: {unknown}'

When it happens

Trigger: Passing a plain-string expression like "a + b" or "foo == 1" where 'a', 'b', 'foo' are not in input_fields, or an expression more complex than atomic/atomic comparisons, to validate_generic_expression without a language-specifying map.

Common situations: Typos in field names; referencing fields that don't exist in the input schema; using arithmetic/complex logic that requires explicitly choosing a language (python/jinja/javascript).

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Ambiguous expression type (perhaps missing quoting?)
• Ambiguous expression type (perhaps missing quoting?)
• Can only use expressions on a schema'd input.
• CombineFn spec missing type
• Config for transform at

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4914416fa1bb2ea7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_mapping.py:135

        "Must specify a language when using a map with custom logic for %s" %
        error_field)
  expr = str(expr_dict['expression'])

  def is_atomic(expr: str):
    return is_literal(expr) or expr in input_fields

  if is_atomic(expr):
    return

  if allow_cmp:
    maybe_cmp = re.fullmatch('(.*)([<>=!]+)(.*)', expr)
    if maybe_cmp:
      left, cmp, right = maybe_cmp.groups()
      if (is_atomic(left.strip()) and is_atomic(right.strip()) and
          cmp in {'==', '<=', '>=', '<', '>', '!='}):
        return

  raise ValueError(
      "Missing language specification, unknown input fields, "
      f"or invalid generic expression: {expr}. "
      f"The given input fields are {input_fields}. "
      "See https://beam.apache.org/documentation/sdks/yaml-udf/#generic")


def validate_generic_expressions(base_type, config, input_pcolls) -> None:
  if not input_pcolls:
    return
  try:
    input_fields = [
        name for (name, _) in named_fields_from_element_type(
            next(iter(input_pcolls)).element_type)
    ]
  except (TypeError, ValueError):
    input_fields = []

  if base_type == 'MapToFields':

View on GitHub (pinned to 12126d8942)