apache/beam · error · ValueError

An invalid input " " was specified in "fields".

Error message

An invalid input "{input}" was specified in "fields".

What it means

While parsing the 'fields' section of a YAML Join spec, _parse_fields encountered a key naming an input that is not among the transform's actual input tables; the named input is a typo or leftover from a renamed input.

Solutions

Use only input aliases declared in the transform's 'input' mapping as fields keys.
Fix or remove the invalid key from fields.
Keep input aliases and fields keys in sync when renaming.

Example fix

// before
input: {left: read_a, right: read_b}
fields:
  lft: [id]
// after
input: {left: read_a, right: read_b}
fields:
  left: [id]

Defensive patterns

Strategy: validation

Validate before calling

def check_fields_keys(fields, input_aliases):
    bad = set(fields) - set(input_aliases)
    if bad:
        raise ValueError(f'fields keys not in inputs: {bad}')

Try / catch

try:
    result = SqlJoinTransform(config)
except ValueError as e:
    if 'was specified in "fields"' in str(e):
        raise ConfigError('fields keys must match declared input aliases') from e
    raise

Prevention

Keep fields keys and input aliases in one source of truth
Search the spec for each fields key to verify it is declared in input
Validate the spec against the input aliases before submitting the job

When it happens

Trigger: A fields key like extra: [...] where 'extra' is not one of the declared input aliases; renaming an input alias without updating fields; a typo in the alias inside fields.

Common situations: Hand-edited YAML with inconsistent alias names; copying the fields block from a different join config with different input names.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f0ad9ffb3b4dd5d6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_join.py:118

    input_edge_list.append(tuple(equality.keys()))

  if not _is_connected(input_edge_list, len(pcolls)):
    raise ValueError(
        f'{error_prefix} '
        f'The provided equalities do not connect all of {list(pcolls.keys())}.')

  return equalities


def _parse_fields(tables, fields):
  error_prefix = f'Invalid value "{fields}" for "fields".'
  if not isinstance(fields, dict):
    raise ValueError(f'{error_prefix} Fields must be a dict.')
  output_fields = []
  named_columns = set()
  for input, cols in fields.items():
    if input not in tables:
      raise ValueError(f'An invalid input "{input}" was specified in "fields".')
    if isinstance(cols, list):
      for col in cols:
        if not isinstance(col, str):
          raise ValueError(
              f'Invalid column "{col}" in "fields". Column name must be a str.')
        if col in named_columns:
          raise ValueError(
              f'The field name "{col}" was specified more than once.')
        output_fields.append(f'{input}.{col} AS {col}')
        named_columns.add(col)
    elif isinstance(cols, dict):
      for k, v in cols.items():
        if k in named_columns:
          raise ValueError(
              f'The field name "{k}" was specified more than once.')
        if not isinstance(v, str):
          raise ValueError(
              f'Invalid column "{v}" in "fields". Column name must be a str.')

View on GitHub (pinned to 12126d8942)