apache/beam · error · ValueError

Dropping unknown field

Error message

Dropping unknown field "{name}"

What it means

Raised by `normalize_fields` when a name listed in `drop` is not a field of the input PCollection's schema. Like Explode's unknown-field check, it validates each dropped name against `named_fields_from_element_type` to fail fast on misspellings or stale column references.

Solutions

1. Correct the field name in the `drop` list to match the input schema exactly (case-sensitive).
2. Log or inspect the input schema and remove nonexistent fields from the drop list.
3. Update upstream transforms if the field was renamed.

Example fix

// before
config:
  append: true
  drop: [temprature]
// after
config:
  append: true
  drop: [temperature]

schema_fields = {n for n, _ in named_fields_from_element_type(pcoll.element_type)}
unknown = set(drop) - schema_fields
if unknown:
    raise ValueError(f'Drop names not in input schema: {unknown}; available: {sorted(schema_fields)}')

try:
    out = normalize_fields(pcoll, fields, drop=['temprature'], append=True)
except ValueError as e:
    if 'Dropping unknown field' in str(e):
        logger.error('Check spelling against schema: %s', e)
    raise

When it happens

Trigger: YAML transform config with `drop: [name]` (and append true) where `name` does not exist in the input schema — typically a typo, or the column was renamed/removed by an upstream step.

Common situations: Renaming a field in an earlier Map step without updating the drop list; case-sensitivity mistakes (Id vs id); dropping a field that only exists in a different branch of the pipeline.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Can only append fields on a schema'd input.
• Can only drop fields on a schema'd input.
• Cannot convert element of type
• Exploding unknown field
• f"Mapping destinations

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/727e312ab1007280. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_mapping.py:678

def normalize_fields(pcoll, fields, drop=(), append=False, language='generic'):
  try:
    input_schema = dict(named_fields_from_element_type(pcoll.element_type))
  except (TypeError, ValueError) as exn:
    if drop:
      raise ValueError("Can only drop fields on a schema'd input.") from exn
    if append:
      raise ValueError("Can only append fields on a schema'd input.") from exn
    elif any(is_expr(x) for x in fields.values()):
      raise ValueError("Can only use expressions on a schema'd input.") from exn
    input_schema = {}

  if drop and not append:
    raise ValueError("Can only drop fields if append is true.")
  for name in drop:
    if name not in input_schema:
      raise ValueError(f'Dropping unknown field "{name}"')
  if append:
    for name in fields:
      if name in input_schema and name not in drop:
        raise ValueError(
            f'Redefinition of field "{name}". '
            'Cannot append a field that already exists in original input.')

  if append:
    return input_schema, {
        **{name: f'`{name}`' if language in ['sql', 'calcite'] else name
           for name in input_schema.keys() if name not in drop},
        **fields
    }
  else:
    return input_schema, fields


@beam.ptransform.ptransform_fn

View on GitHub (pinned to 12126d8942)