apache/beam · error · WontImplementError

() of non-categorical type is not supported because the…

Error message

{method_name}() of non-categorical type is not supported because the type of the output column depends on the data. Please use pd.CategoricalDtype with explicit categories.

What it means

str.split/str.rsplit with expand=True (separate output columns) raises WontImplementError when the series dtype is not CategoricalDtype. When splitting into distinct columns, the column names and count depend on the categories found in the data; Beam requires a categorical dtype so the resulting schema is static.

Solutions

Cast to categorical first: s.astype(pd.CategoricalDtype(categories=[...])) before calling split/rsplit with expand=True.
Use expand=False (default) to get a list-like single column, then extract parts with .str[i] into named columns.
Create each output column explicitly with .str.split(...).str.get(0), .str.get(1), etc.
Do the expand split in plain pandas outside the pipeline.

Example fix

// before
parts = s.str.split('-', expand=True)

// after
parts0 = s.str.split('-').str.get(0)
parts1 = s.str.split('-').str.get(1)

Defensive patterns

Strategy: validation

Validate before calling

if expand and not isinstance(s._expr.proxy().dtype, pd.CategoricalDtype):
    raise ValueError("cast to CategoricalDtype before split(expand=True)")

Type guard

def is_categorical(s):
    return isinstance(s._expr.proxy().dtype, pd.CategoricalDtype)

Try / catch

try:
    parts = s.str.split(sep, expand=True)
except apachebeam.WontImplementError:
    parts0 = s.str.split(sep).str.get(0)
    parts1 = s.str.split(sep).str.get(1)

Prevention

Prefer expand=False plus .str.get(i) column extraction in Beam.
Declare categories via astype(pd.CategoricalDtype(...)) when expand=True is needed.
Check the proxy dtype with s.dtype at pipeline-construction time.

When it happens

Trigger: s.str.split(sep, expand=True) or s.str.rsplit(sep, expand=True) where s.dtype is not pd.CategoricalDtype on a deferred Beam Series (the error text names 'split' or 'rsplit' via method_name).

Common situations: Splitting a delimited field into multiple DataFrame columns in a Beam pipeline; porting pandas expand=True splits; forgetting that Beam requires declared categories for column-creating operations.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9b77b9e87478c293. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:5146

            requires_partition_by=partitionings.Arbitrary(),
            preserves_partition_by=partitionings.Arbitrary()))

  def _split_helper(self, rsplit=False, **kwargs):
    expand = kwargs.get('expand', False)

    if not expand:
      # Not creating separate columns
      proxy = self._expr.proxy()
      if not rsplit:
        func = lambda s: pd.concat([proxy, s.str.split(**kwargs)])
      else:
        func = lambda s: pd.concat([proxy, s.str.rsplit(**kwargs)])
    else:
      # Creating separate columns, so must be more strict on dtype
      dtype = self._expr.proxy().dtype
      if not isinstance(dtype, pd.CategoricalDtype):
        method_name = 'rsplit' if rsplit else 'split'
        raise frame_base.WontImplementError(
            f"{method_name}() of non-categorical type is not supported because "
            "the type of the output column depends on the data. Please use "
            "pd.CategoricalDtype with explicit categories.",
            reason="non-deferred-columns")

      # Split the categories
      split_cats = dtype.categories.str.split(**kwargs)

      # Count the number of new columns to create for proxy
      max_splits = len(max(split_cats, key=len))
      proxy = pd.DataFrame(columns=range(max_splits))

      def func(s):
        if not rsplit:
          result = s.str.split(**kwargs)
        else:
          result = s.str.rsplit(**kwargs)
        result[~result.isna()].replace(np.nan, value=None)

View on GitHub (pinned to 12126d8942)