apache/beam · error · ValueError

Can only drop fields if append is true.

Error message

Can only drop fields if append is true.

What it means

Raised by `normalize_fields` when `drop` is provided without `append: true`. In Beam YAML's field normalization, dropping fields is only meaningful as part of an append (redefine-and-drop) operation; dropping alone is rejected to force the caller to state intent explicitly.

Solutions

1. Add `append: true` to the transform config when using `drop`.
2. Use the dedicated DropFields YAML transform instead if you only want to remove fields.
3. Remove the `drop` option if dropping was not intended.

Example fix

// before
- type: AddFields
  input: rows
  config:
    drop: [temp_col]
// after
- type: AddFields
  input: rows
  config:
    append: true
    drop: [temp_col]
    fields: {}

if drop and not append:
    raise ValueError('drop requires append: true in this transform; or use DropFields instead.')

try:
    out = normalize_fields(pcoll, fields={}, drop=['temp'])
except ValueError as e:
    if 'drop fields if append is true' in str(e):
        out = normalize_fields(pcoll, fields={}, drop=['temp'], append=True)
    else:
        raise

When it happens

Trigger: Invoking a YAML mapping transform with `drop: [field]` but no `append: true` in the config, after the schema lookup succeeded (input is schema'd). The check `if drop and not append` fires.

Common situations: Confusing the DropFields-like semantics: users expect a standalone drop, but this transform requires append alongside drop; copy-pasted configs missing the append flag.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Error handling output
• Exploding unknown field
• Input schema has multiple fields
• Invalid feature store yaml file provided. Make sure the
• Only expressions allowed in SQL at

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/890526e09129ab96. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_mapping.py:675

def is_expr(v):
  return isinstance(v, str) or (isinstance(v, dict) and 'expression' in v)


def normalize_fields(pcoll, fields, drop=(), append=False, language='generic'):
  try:
    input_schema = dict(named_fields_from_element_type(pcoll.element_type))
  except (TypeError, ValueError) as exn:
    if drop:
      raise ValueError("Can only drop fields on a schema'd input.") from exn
    if append:
      raise ValueError("Can only append fields on a schema'd input.") from exn
    elif any(is_expr(x) for x in fields.values()):
      raise ValueError("Can only use expressions on a schema'd input.") from exn
    input_schema = {}

  if drop and not append:
    raise ValueError("Can only drop fields if append is true.")
  for name in drop:
    if name not in input_schema:
      raise ValueError(f'Dropping unknown field "{name}"')
  if append:
    for name in fields:
      if name in input_schema and name not in drop:
        raise ValueError(
            f'Redefinition of field "{name}". '
            'Cannot append a field that already exists in original input.')

  if append:
    return input_schema, {
        **{name: f'`{name}`' if language in ['sql', 'calcite'] else name
           for name in input_schema.keys() if name not in drop},
        **fields
    }
  else:
    return input_schema, fields

View on GitHub (pinned to 12126d8942)