apache/beam · error · ValueError

Redefinition of field

Error message

Redefinition of field "{name}". Cannot append a field that already exists in original input.

What it means

Raised by `normalize_fields` when `append: true` and the `fields` mapping tries to add a field whose name already exists in the input schema and is not being dropped. Appending is strictly additive; to overwrite an existing column you must also list it in `drop` (append-and-redefine) rather than silently replacing it.

Solutions

1. Add the existing field name to the `drop` list so it is redefined rather than duplicated.
2. Rename the new field to a non-conflicting name.
3. Remove the conflicting field from `fields` if the existing column is already correct.

Example fix

// before
config:
  append: true
  fields:
    score: {type: double, value: "score * 2"}
// after
config:
  append: true
  drop: [score]
  fields:
    score: {type: double, value: "score * 2"}

schema_fields = dict(named_fields_from_element_type(pcoll.element_type)).keys()
conflicts = [n for n in fields if n in schema_fields and n not in drop]
if conflicts:
    raise ValueError(f'Appending existing fields without dropping them: {conflicts}; add them to drop.')

try:
    out = normalize_fields(pcoll, fields={'score': expr}, drop=[], append=True)
except ValueError as e:
    if 'Redefinition of field' in str(e):
        out = normalize_fields(pcoll, fields={'score': expr}, drop=['score'], append=True)
    else:
        raise

When it happens

Trigger: YAML AddFields/MapToFields config with `append: true` and a `fields` key that matches an existing input column, where that column is absent from the `drop` list. The check `if name in input_schema and name not in drop` fires.

Common situations: Attempting to overwrite/replace an existing column via AddFields; unaware that redefining requires drop; copy-pasting configs where the added field already exists upstream.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Exploding unknown field
• Input schema has multiple fields
• Can only append fields on a schema'd input.
• Can only drop fields if append is true.
• Can only drop fields on a schema'd input.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2bef2658f5b4acff. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_mapping.py:682

    input_schema = dict(named_fields_from_element_type(pcoll.element_type))
  except (TypeError, ValueError) as exn:
    if drop:
      raise ValueError("Can only drop fields on a schema'd input.") from exn
    if append:
      raise ValueError("Can only append fields on a schema'd input.") from exn
    elif any(is_expr(x) for x in fields.values()):
      raise ValueError("Can only use expressions on a schema'd input.") from exn
    input_schema = {}

  if drop and not append:
    raise ValueError("Can only drop fields if append is true.")
  for name in drop:
    if name not in input_schema:
      raise ValueError(f'Dropping unknown field "{name}"')
  if append:
    for name in fields:
      if name in input_schema and name not in drop:
        raise ValueError(
            f'Redefinition of field "{name}". '
            'Cannot append a field that already exists in original input.')

  if append:
    return input_schema, {
        **{name: f'`{name}`' if language in ['sql', 'calcite'] else name
           for name in input_schema.keys() if name not in drop},
        **fields
    }
  else:
    return input_schema, fields


@beam.ptransform.ptransform_fn
@maybe_with_exception_handling_transform_fn
def _PyJsMapToFields(
    pcoll,
    fields: Mapping[str, Union[str, Mapping[str, str]]],

View on GitHub (pinned to 12126d8942)