{"record":{"id":"965efb59958d284c","repo":"apache/beam","slug":"can-only-use-expressions-on-a-schema-d-input","errorCode":null,"errorMessage":"Can only use expressions on a schema'd input.","messagePattern":"Can only use expressions on a schema'd input\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_mapping.py","lineNumber":381,"sourceCode":"        validator(getattr(row, name))\n        for (name, validator) in validators.items())\n  else:\n    raise ValueError(f\"Unrecognized type_info: {type_info!r}\")\n\n\ndef _as_callable_for_pcoll(\n    pcoll,\n    fn_spec: Union[str, dict[str, str]],\n    msg: str,\n    language: Optional[str]):\n  if language == 'javascript':\n    options.YamlOptions.check_enabled(pcoll.pipeline, 'javascript')\n\n  try:\n    input_schema = dict(named_fields_from_element_type(pcoll.element_type))\n  except (TypeError, ValueError) as exn:\n    if is_expr(fn_spec):\n      raise ValueError(\"Can only use expressions on a schema'd input.\") from exn\n    input_schema = {}  # unused\n\n  if isinstance(fn_spec, str) and fn_spec in input_schema:\n    return lambda row: getattr(row, fn_spec)\n  else:\n    return _as_callable(\n        list(input_schema.keys()), fn_spec, msg, language, input_schema)\n\n\ndef _as_callable(original_fields, expr, transform_name, language, input_schema):\n  if isinstance(expr, str):\n    expr = {'expression': expr}\n\n  # Extract original type from upstream pcoll when doing simple mappings\n  original_type = input_schema.get(expr.get('expression'), None)\n  if expr in original_fields:\n    language = \"python\"\n","sourceCodeStart":363,"sourceCodeEnd":399,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_mapping.py#L363-L399","documentation":"In `_as_callable_for_pcoll`, if the input PCollection's element type cannot be turned into a named schema (no schema), expressions—which need to reference named fields—cannot work. When the fn_spec looks like an expression, the transform raises this ValueError instead of silently proceeding.","triggerScenarios":"Using MapToFields (or Filter/Partition/AssignTimestamps via _as_callable_for_pcoll) with an `expression:` config on a PCollection whose elements are untyped (e.g. plain strings, ints, or dicts rather than schema'd rows), such as output of a raw Read or non-schema transform.","commonSituations":"Applying a YAML MapToFields transform directly after a text/byte source like ReadFromText, or after a transform that loses the schema, while using field-referencing expressions.","solutions":["Ensure the input has a schema: insert a parse/convert step (e.g. json.Parse or a MapToFields with explicit output_type defining the fields) before the expression-based transform.","If the input is genuinely untyped, use a plain callable/lambda (`callable:` or `path:` config) instead of an `expression:`.","Verify the upstream transform preserves the Beam schema (check with a Print or schema inspection)."],"exampleFix":"- type: MapToFields\n  input: RawText   # strings, no schema -> fails\n# after: parse first\n- type: MapToFields\n  input: ParsedJson\n  config:\n    input_type:\n      id: string\n    fields:\n      id: \"id.upper()\"","handlingStrategy":"validation","validationCode":"# ensure input has a schema before using expressions\nfrom apache_beam.typehints.schemas import named_fields_from_element_type\nfields = named_fields_from_element_type(pcoll.element_type)  # raises if untyped","typeGuard":"def is_schema_apcoll(pcoll) -> bool:\n    try:\n        named_fields_from_element_type(pcoll.element_type)\n        return True\n    except (TypeError, ValueError):\n        return False","tryCatchPattern":"try:\n    out = apply_map_to_fields(pcoll, cfg)\nexcept ValueError as e:\n    if \"expressions on a schema'd input\" in str(e):\n        pcoll = parse_to_schema(pcoll)\n        out = apply_map_to_fields(pcoll, cfg)\n    else:\n        raise","preventionTips":["Insert a parse step (e.g. JSON parse) before expression-based transforms on raw text","Check the input PCollection has named schema fields","Use callable/path UDFs instead of expressions when input is untyped"],"tags":["yaml","beam-yaml","schema","expressions"],"backgroundTag":"incompatible-source-type","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}