{"record":{"id":"a72255b4619768da","repo":"apache/beam","slug":"can-only-drop-fields-on-a-schema-d-input","errorCode":null,"errorMessage":"Can only drop fields on a schema'd input.","messagePattern":"Can only drop fields on a schema'd input\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_mapping.py","lineNumber":667,"sourceCode":"    language: The language of the above expression.\n      Defaults to generic.\n    error_handling: Whether and where to output records that throw errors when\n      the above expressions are evaluated.\n  \"\"\"  # pylint: disable=line-too-long\n  keep_fn = _as_callable_for_pcoll(pcoll, keep, \"keep\", language or 'generic')\n  return pcoll | beam.Filter(keep_fn)\n\n\ndef is_expr(v):\n  return isinstance(v, str) or (isinstance(v, dict) and 'expression' in v)\n\n\ndef normalize_fields(pcoll, fields, drop=(), append=False, language='generic'):\n  try:\n    input_schema = dict(named_fields_from_element_type(pcoll.element_type))\n  except (TypeError, ValueError) as exn:\n    if drop:\n      raise ValueError(\"Can only drop fields on a schema'd input.\") from exn\n    if append:\n      raise ValueError(\"Can only append fields on a schema'd input.\") from exn\n    elif any(is_expr(x) for x in fields.values()):\n      raise ValueError(\"Can only use expressions on a schema'd input.\") from exn\n    input_schema = {}\n\n  if drop and not append:\n    raise ValueError(\"Can only drop fields if append is true.\")\n  for name in drop:\n    if name not in input_schema:\n      raise ValueError(f'Dropping unknown field \"{name}\"')\n  if append:\n    for name in fields:\n      if name in input_schema and name not in drop:\n        raise ValueError(\n            f'Redefinition of field \"{name}\". '\n            'Cannot append a field that already exists in original input.')\n","sourceCodeStart":649,"sourceCodeEnd":685,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_mapping.py#L649-L685","documentation":"Raised by `normalize_fields` when the `drop` option is supplied but the input PCollection's element type is not a Beam schema (calling `named_fields_from_element_type` raises TypeError/ValueError). Dropping fields requires an existing schema to know which columns to remove, so without one the transform cannot proceed and raises instead.","triggerScenarios":"Using a Map/AddFields/DropFields-style YAML transform with `drop: [field]` where the input PCollection has no schema attached (e.g. output of a generic Map, JSON parse, or DoFn without type hints). The `drop` argument is truthy and the schema lookup fails.","commonSituations":"Pipeline reads untyped JSON or a DoFn emits plain dicts without `with_output_types`; connecting a drop step directly to a non-row source; missing schema inference after a Python map lambda.","solutions":["Attach a schema to the input PCollection (e.g. `beam.Map(lambda x: beam.Row(**x))` with proper types or a Cast transform in YAML).","Remove the `drop` option if the input truly has no fields to drop.","Use a Map transform with explicit expressions instead of drop for untyped inputs."],"exampleFix":"// before\n- type: AddFields\n  input: generic_map_output\n  config:\n    drop: [temp]\n// after\n- type: Cast\n  input: generic_map_output\n  config:\n    fields: {id: int64, name: string, temp: string}\n- type: AddFields\n  input: cast_output\n  config:\n    drop: [temp]","handlingStrategy":"validation","validationCode":"from apache_beam.yaml.yaml_mapping import named_fields_from_element_type\ntry:\n    named_fields_from_element_type(pcoll.element_type)\nexcept (TypeError, ValueError):\n    raise ValueError('Input to drop-fields step has no schema; insert a Cast step first.')","typeGuard":null,"tryCatchPattern":"try:\n    result = normalize_fields(pcoll, fields={}, drop=['temp'])\nexcept ValueError as e:\n    if \"schema'd input\" in str(e):\n        pcoll = cast_to_schema(pcoll)  # insert a schema step\n    else:\n        raise","preventionTips":["Always give PCollections explicit schemas (beam.Row with type hints) before YAML field transforms","Avoid feeding raw dict outputs of lambdas into schema-dependent transforms","Insert Cast steps after untyped sources (JSON, generic maps) as a habit"],"tags":["apache-beam","yaml","schema"],"backgroundTag":"incompatible-source-type","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}