What causes "The field name "{col}" was specified more than once." in apache/beam?

Use the dict form to rename one occurrence: right: {right_id: id}.

apache/beam · error · ValueError

The field name " " was specified more than once.

Error message

The field name "{col}" was specified more than once.

What it means

Output field names must be unique across the joined result. When a list-style columns entry names a column that has already been emitted by another input (tracked in named_columns), _parse_fields raises this duplicate-name ValueError to avoid ambiguous SELECT output.

Solutions

Use the dict form to rename one occurrence: right: {right_id: id}.
Remove the duplicate column from one input's list.
Exclude the conflicting column and rely on the default table.* expansion carefully, renaming upstream.

Example fix

// before
fields:
  left: [id]
  right: [id]
// after
fields:
  left: [id]
  right: {right_id: id}

Defensive patterns

Strategy: validation

Validate before calling

def check_unique_names(fields):
    seen = set()
    for alias, cols in fields.items():
        names = cols if isinstance(cols, list) else (cols.keys() if isinstance(cols, dict) else [])
        for n in names:
            if n in seen:
                raise ValueError(f'duplicate output field: {n}')
            seen.add(n)

Try / catch

try:
    result = SqlJoinTransform(config)
except ValueError as e:
    if 'specified more than once' in str(e):
        raise ConfigError('rename one of the duplicate output fields with dict form') from e
    raise

Prevention

Use dict form {outName: col} to rename shared columns like id
List output names across inputs before finalizing the spec
Prefer explicit renaming for join-key columns with common names

When it happens

Trigger: Two inputs both contributing a column named 'id' via list form, e.g. fields: {left: [id], right: [id]}.

Common situations: Joins between tables that share key column names (id, key, name); users forgetting that list form emits the raw column name, unlike dict form which allows renaming.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c50f821e4951b09d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_join.py:125

  return equalities


def _parse_fields(tables, fields):
  error_prefix = f'Invalid value "{fields}" for "fields".'
  if not isinstance(fields, dict):
    raise ValueError(f'{error_prefix} Fields must be a dict.')
  output_fields = []
  named_columns = set()
  for input, cols in fields.items():
    if input not in tables:
      raise ValueError(f'An invalid input "{input}" was specified in "fields".')
    if isinstance(cols, list):
      for col in cols:
        if not isinstance(col, str):
          raise ValueError(
              f'Invalid column "{col}" in "fields". Column name must be a str.')
        if col in named_columns:
          raise ValueError(
              f'The field name "{col}" was specified more than once.')
        output_fields.append(f'{input}.{col} AS {col}')
        named_columns.add(col)
    elif isinstance(cols, dict):
      for k, v in cols.items():
        if k in named_columns:
          raise ValueError(
              f'The field name "{k}" was specified more than once.')
        if not isinstance(v, str):
          raise ValueError(
              f'Invalid column "{v}" in "fields". Column name must be a str.')
        output_fields.append(f'{input}.{v} AS {k}')
        named_columns.add(k)
    else:
      raise ValueError(
          f'{error_prefix} '
          f'For every input key in fields, '
          f'the value must either be a list or dict.')

View on GitHub (pinned to 12126d8942)