apache/beam · error · ValueError

Invalid value " " for "fields". Fields must be a dict.

Error message

Invalid value "{fields}" for "fields". Fields must be a dict.

What it means

The optional 'fields' parameter of the SQL Join transform selects which columns each input contributes to the output. _parse_fields requires it to be a mapping from input alias to a list or dict of columns; if 'fields' is not a dict this ValueError is raised.

Solutions

Make fields a mapping of input alias -> list of columns (or alias -> {outputName: columnName} dict).
Quote/indent YAML correctly so the value parses as a dict.
Remove the fields key entirely to include all columns from all inputs.

Example fix

// before
fields: [id, name]
// after
fields:
  left: [id, name]
  right: [id, name]

Defensive patterns

Strategy: type-guard

Validate before calling

if not isinstance(spec.get('fields'), dict):
    raise ValueError('fields must be a mapping of input alias -> columns')

Type guard

def has_valid_fields_shape(spec):
    f = spec.get('fields')
    return f is None or (isinstance(f, dict) and all(isinstance(v, (list, dict)) for v in f.values()))

Try / catch

try:
    result = SqlJoinTransform(config)
except ValueError as e:
    if 'Fields must be a dict' in str(e):
        raise ConfigError('fields must map input aliases to column lists/dicts') from e
    raise

Prevention

Remember fields is per-input: alias -> list-or-dict
Check YAML indentation so fields parses as a mapping
Use a JSON/YAML schema validator on the pipeline spec

When it happens

Trigger: Passing fields as a list (e.g. fields: [id, name]), a string (fields: id), or null/other non-dict value in the Join transform spec.

Common situations: Users assuming fields is a flat list of column names across all inputs; YAML indentation mistakes that turn the mapping into a list; copying config from a different transform with a different fields schema.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9cc85b1e223bd781. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_join.py:113

            f'{error_prefix} "{pcoll_tag}" is not a specified alias in "input"')
      if col not in valid_cols[pcoll_tag]:
        raise ValueError(
            f'{error_prefix} "{col}" is not a valid field in "{pcoll_tag}".')

    input_edge_list.append(tuple(equality.keys()))

  if not _is_connected(input_edge_list, len(pcolls)):
    raise ValueError(
        f'{error_prefix} '
        f'The provided equalities do not connect all of {list(pcolls.keys())}.')

  return equalities


def _parse_fields(tables, fields):
  error_prefix = f'Invalid value "{fields}" for "fields".'
  if not isinstance(fields, dict):
    raise ValueError(f'{error_prefix} Fields must be a dict.')
  output_fields = []
  named_columns = set()
  for input, cols in fields.items():
    if input not in tables:
      raise ValueError(f'An invalid input "{input}" was specified in "fields".')
    if isinstance(cols, list):
      for col in cols:
        if not isinstance(col, str):
          raise ValueError(
              f'Invalid column "{col}" in "fields". Column name must be a str.')
        if col in named_columns:
          raise ValueError(
              f'The field name "{col}" was specified more than once.')
        output_fields.append(f'{input}.{col} AS {col}')
        named_columns.add(col)
    elif isinstance(cols, dict):
      for k, v in cols.items():
        if k in named_columns:

View on GitHub (pinned to 12126d8942)