apache/beam · error · ValueError

Can only drop fields on a schema'd input.

Error message

Can only drop fields on a schema'd input.

What it means

Raised by `normalize_fields` when the `drop` option is supplied but the input PCollection's element type is not a Beam schema (calling `named_fields_from_element_type` raises TypeError/ValueError). Dropping fields requires an existing schema to know which columns to remove, so without one the transform cannot proceed and raises instead.

Solutions

1. Attach a schema to the input PCollection (e.g. `beam.Map(lambda x: beam.Row(**x))` with proper types or a Cast transform in YAML).
2. Remove the `drop` option if the input truly has no fields to drop.
3. Use a Map transform with explicit expressions instead of drop for untyped inputs.

Example fix

// before
- type: AddFields
  input: generic_map_output
  config:
    drop: [temp]
// after
- type: Cast
  input: generic_map_output
  config:
    fields: {id: int64, name: string, temp: string}
- type: AddFields
  input: cast_output
  config:
    drop: [temp]

from apache_beam.yaml.yaml_mapping import named_fields_from_element_type
try:
    named_fields_from_element_type(pcoll.element_type)
except (TypeError, ValueError):
    raise ValueError('Input to drop-fields step has no schema; insert a Cast step first.')

try:
    result = normalize_fields(pcoll, fields={}, drop=['temp'])
except ValueError as e:
    if "schema'd input" in str(e):
        pcoll = cast_to_schema(pcoll)  # insert a schema step
    else:
        raise

When it happens

Trigger: Using a Map/AddFields/DropFields-style YAML transform with `drop: [field]` where the input PCollection has no schema attached (e.g. output of a generic Map, JSON parse, or DoFn without type hints). The `drop` argument is truthy and the schema lookup fails.

Common situations: Pipeline reads untyped JSON or a DoFn emits plain dicts without `with_output_types`; connecting a drop step directly to a non-row source; missing schema inference after a Python map lambda.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• Can only append fields on a schema'd input.
• Cannot convert element of type
• Dropping unknown field
• Exploding unknown field
• f"Mapping destinations

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a72255b4619768da. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_mapping.py:667

    language: The language of the above expression.
      Defaults to generic.
    error_handling: Whether and where to output records that throw errors when
      the above expressions are evaluated.
  """  # pylint: disable=line-too-long
  keep_fn = _as_callable_for_pcoll(pcoll, keep, "keep", language or 'generic')
  return pcoll | beam.Filter(keep_fn)


def is_expr(v):
  return isinstance(v, str) or (isinstance(v, dict) and 'expression' in v)


def normalize_fields(pcoll, fields, drop=(), append=False, language='generic'):
  try:
    input_schema = dict(named_fields_from_element_type(pcoll.element_type))
  except (TypeError, ValueError) as exn:
    if drop:
      raise ValueError("Can only drop fields on a schema'd input.") from exn
    if append:
      raise ValueError("Can only append fields on a schema'd input.") from exn
    elif any(is_expr(x) for x in fields.values()):
      raise ValueError("Can only use expressions on a schema'd input.") from exn
    input_schema = {}

  if drop and not append:
    raise ValueError("Can only drop fields if append is true.")
  for name in drop:
    if name not in input_schema:
      raise ValueError(f'Dropping unknown field "{name}"')
  if append:
    for name in fields:
      if name in input_schema and name not in drop:
        raise ValueError(
            f'Redefinition of field "{name}". '
            'Cannot append a field that already exists in original input.')

View on GitHub (pinned to 12126d8942)