apache/beam · error · WontImplementError

astype(dtype='category') is not supported because the type…

Error message

astype(dtype='category') is not supported because the type of the output column depends on the data. Please use pd.CategoricalDtype with explicit categories instead.

What it means

astype('category') is rejected because the resulting categorical categories are derived from the data itself, which would make the output column types data-dependent — a non-deferred property the Beam DataFrame API forbids (columns must not depend on data values). Passing an explicit pd.CategoricalDtype with fixed categories is allowed.

Solutions

Use an explicit dtype: df.astype(pd.CategoricalDtype(categories=['a','b','c'])).
Derive the category list from a separate (bounded, collected) pass, then build pd.CategoricalDtype from it.
Keep the categorical conversion in a to_pandas() stage instead.
Use map/replace to encode values to fixed codes rather than categorical dtype.

Example fix

// before
df = df.astype('category')
// after
df = df.astype(pd.CategoricalDtype(categories=['low', 'medium', 'high']))

Defensive patterns

Strategy: validation

Validate before calling

if dtype == 'category':
    raise ValueError('Use pd.CategoricalDtype(categories=[...]) instead of the string category')

Type guard

def is_explicit_categorical(dtype) -> bool:
    return isinstance(dtype, pd.CategoricalDtype)

Try / catch

try:
    df = df.astype('category')
except frame_base.WontImplementError:
    df = df.astype(pd.CategoricalDtype(categories=['a', 'b', 'c']))

Prevention

Never astype to the bare string 'category' in Beam
Define category sets from config or a prior collection pass
Prefer fixed code mappings (map/replace) over inferred categoricals

When it happens

Trigger: df.astype('category') or series.astype('category') with the plain string 'category' (and not a pd.CategoricalDtype instance) on a deferred frame.

Common situations: Encoding columns to categorical dtype for memory savings or ML preprocessing; pandas ports that relied on implicit category inference; string-to-category conversions in ETL.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d8ea8fa5ce15e7a6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:647

      # data.
      requires = partitionings.Singleton(
          reason=(
              f"astype(errors={errors!r}) is currently not parallelizable, "
              "because all data must be collected on one node to determine if "
              "the original data should be propagated instead."))

    if not copy:
      raise frame_base.WontImplementError(
          f"astype(copy={copy!r}) is not supported because it relies on "
          "memory-sharing semantics that are not compatible with the Beam "
          "model.")

    # An instance of CategoricalDtype is actualy considered equal to the string
    # 'category', so we have to explicitly check if dtype is an instance of
    # CategoricalDtype, and allow it.
    # See https://github.com/apache/beam/issues/23276
    if dtype == 'category' and not isinstance(dtype, pd.CategoricalDtype):
      raise frame_base.WontImplementError(
          "astype(dtype='category') is not supported because the type of the "
          "output column depends on the data. Please use pd.CategoricalDtype "
          "with explicit categories instead.",
          reason="non-deferred-columns")

    return frame_base.DeferredFrame.wrap(
        expressions.ComputedExpression(
            'astype',
            lambda df: df.astype(dtype=dtype, copy=copy, errors=errors),
            [self._expr],
            requires_partition_by=requires,
            preserves_partition_by=partitionings.Arbitrary()))

  at_time = frame_base._elementwise_method(
      'at_time', base=pd.core.generic.NDFrame)
  between_time = frame_base._elementwise_method(
      'between_time', base=pd.core.generic.NDFrame)
  copy = frame_base._elementwise_method('copy', base=pd.core.generic.NDFrame)

View on GitHub (pinned to 12126d8942)