{"record":{"id":"727e312ab1007280","repo":"apache/beam","slug":"dropping-unknown-field-name","errorCode":null,"errorMessage":"Dropping unknown field \"{name}\"","messagePattern":"Dropping unknown field \"(.+?)\"","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_mapping.py","lineNumber":678,"sourceCode":"\n\ndef normalize_fields(pcoll, fields, drop=(), append=False, language='generic'):\n  try:\n    input_schema = dict(named_fields_from_element_type(pcoll.element_type))\n  except (TypeError, ValueError) as exn:\n    if drop:\n      raise ValueError(\"Can only drop fields on a schema'd input.\") from exn\n    if append:\n      raise ValueError(\"Can only append fields on a schema'd input.\") from exn\n    elif any(is_expr(x) for x in fields.values()):\n      raise ValueError(\"Can only use expressions on a schema'd input.\") from exn\n    input_schema = {}\n\n  if drop and not append:\n    raise ValueError(\"Can only drop fields if append is true.\")\n  for name in drop:\n    if name not in input_schema:\n      raise ValueError(f'Dropping unknown field \"{name}\"')\n  if append:\n    for name in fields:\n      if name in input_schema and name not in drop:\n        raise ValueError(\n            f'Redefinition of field \"{name}\". '\n            'Cannot append a field that already exists in original input.')\n\n  if append:\n    return input_schema, {\n        **{name: f'`{name}`' if language in ['sql', 'calcite'] else name\n           for name in input_schema.keys() if name not in drop},\n        **fields\n    }\n  else:\n    return input_schema, fields\n\n\n@beam.ptransform.ptransform_fn","sourceCodeStart":660,"sourceCodeEnd":696,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_mapping.py#L660-L696","documentation":"Raised by `normalize_fields` when a name listed in `drop` is not a field of the input PCollection's schema. Like Explode's unknown-field check, it validates each dropped name against `named_fields_from_element_type` to fail fast on misspellings or stale column references.","triggerScenarios":"YAML transform config with `drop: [name]` (and append true) where `name` does not exist in the input schema — typically a typo, or the column was renamed/removed by an upstream step.","commonSituations":"Renaming a field in an earlier Map step without updating the drop list; case-sensitivity mistakes (Id vs id); dropping a field that only exists in a different branch of the pipeline.","solutions":["Correct the field name in the `drop` list to match the input schema exactly (case-sensitive).","Log or inspect the input schema and remove nonexistent fields from the drop list.","Update upstream transforms if the field was renamed."],"exampleFix":"// before\nconfig:\n  append: true\n  drop: [temprature]\n// after\nconfig:\n  append: true\n  drop: [temperature]","handlingStrategy":"validation","validationCode":"schema_fields = {n for n, _ in named_fields_from_element_type(pcoll.element_type)}\nunknown = set(drop) - schema_fields\nif unknown:\n    raise ValueError(f'Drop names not in input schema: {unknown}; available: {sorted(schema_fields)}')","typeGuard":null,"tryCatchPattern":"try:\n    out = normalize_fields(pcoll, fields, drop=['temprature'], append=True)\nexcept ValueError as e:\n    if 'Dropping unknown field' in str(e):\n        logger.error('Check spelling against schema: %s', e)\n    raise","preventionTips":["Copy field names from the schema definition rather than typing them","Grep the pipeline spec for the field name to confirm where it is produced","Watch for case sensitivity mismatches between steps"],"tags":["apache-beam","yaml","schema"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}