apache/beam · error · ValueError

Invalid value for yield_elements argument, '%s'. Allowed val

Error message

Invalid value for yield_elements argument, '%s'. Allowed values are 'pandas' and 'schemas'

What it means

convert.to_pcollection accepts yield_elements only as 'pandas' (yield deferred pandas objects) or 'schemas' (yield Beam schema rows). Any other string raises ValueError listing the allowed values. This is a straightforward argument validation to fail fast on typos like 'schema'.

Source

Thrown at sdks/python/apache_beam/dataframe/convert.py:211

        conversion transform.
    always_return_tuple: (optional, default: False) If true, always return
        a tuple of PCollections, even if there's only one output.
    yield_elements: (optional, default: "schemas") If set to "pandas", return
        PCollections containing the raw Pandas objects (DataFrames or Series),
        if set to "schemas", return an element-wise PCollection, where DataFrame
        and Series instances are expanded to one element per row. DataFrames are
        converted to schema-aware PCollections, where column values can be
        accessed by attribute.
    include_indexes: (optional, default: False) When yield_elements="schemas",
        if include_indexes=True, attempt to include index columns in the output
        schema for expanded DataFrames. Raises an error if any of the index
        levels are unnamed (name=None), or if any of the names are not unique
        among all column and index names.
    pipeline: (optional, unless non-deferred dataframes are passed) Used when
        creating a PCollection from a non-deferred dataframe.
  """
  if not yield_elements in ("pandas", "schemas"):
    raise ValueError(
        "Invalid value for yield_elements argument, '%s'. "
        "Allowed values are 'pandas' and 'schemas'" % yield_elements)
  if label is None:
    # Attempt to come up with a reasonable, stable label by retrieving the name
    # of these variables in the calling context.
    label = 'ToPCollection(%s)' % ', '.join(_var_name(e, 3) for e in dataframes)

  # Support for non-deferred dataframes.
  deferred_dataframes = []
  for ix, df in enumerate(dataframes):
    if isinstance(df, frame_base.DeferredBase):
      # TODO(robertwb): Maybe extract pipeline object?
      deferred_dataframes.append(df)
    elif isinstance(df, (pd.Series, pd.DataFrame)):
      if pipeline is None:
        raise ValueError(
            'Pipeline keyword required for non-deferred dataframe conversion.')
      deferred = pipeline | '%s_Defer%s' % (label, ix) >> beam.Create([df])

View on GitHub (pinned to 12126d8942)

Solutions

Use exactly yield_elements='pandas' or 'schemas'.
Validate/normalize config values against the allowed set before calling.
Search code for other spellings ('schema', 'rows') and correct them.
Add an upstream enum/choices check in your config layer.

Example fix

// before
turned = convert.to_pcollection(df, yield_elements='schema')
// after
turned = convert.to_pcollection(df, yield_elements='schemas')

Defensive patterns

Strategy: validation

Validate before calling

ALLOWED = ('pandas', 'schemas')
if yield_elements not in ALLOWED:
    raise ValueError(f"yield_elements must be one of {ALLOWED}, got {yield_elements!r}")

Try / catch

try:
    out = convert.to_pcollection(df, yield_elements=mode)
except ValueError as e:
    if 'Invalid value for yield_elements' in str(e):
        mode = {'schema': 'schemas', 'rows': 'schemas'}.get(mode, 'schemas')
        out = convert.to_pcollection(df, yield_elements=mode)
    else:
        raise

Prevention

Define yield_elements as an Enum/Literal in your config layer.
Watch for singular/plural typos ('schema' vs 'schemas').
Add a choices check where values come from user input.
Use keyword arguments explicitly at call sites.

When it happens

Trigger: Calling convert.to_pcollection(df, yield_elements='schema') (singular typo), yield_elements=True/None, or programmatic strings from config that aren't exactly 'pandas' or 'schemas'.

Common situations: Typo in the enum string; config-driven pipelines passing user-supplied values unvalidated; older code written against an internal variant of the API.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/229b89354680ee78. Report an issue: GitHub.