apache/beam · error · WontImplementError

astype(copy= ) is not supported because it relies on…

Error message

astype(copy={copy!r}) is not supported because it relies on memory-sharing semantics that are not compatible with the Beam model.

What it means

astype(copy=False) asks pandas to avoid copying and share memory with the original data. Beam's distributed execution model has no shared-memory semantics between workers, so this promise cannot be honored, and the API raises WontImplementError.

Solutions

Pass copy=True (or omit copy) when calling astype.
If using pandas >= 1.3 where copy defaults differently, explicitly set copy=True to be safe.
Only take astype(copy=False) after converting to real pandas via to_pandas().

Example fix

// before
df = df.astype('int64', copy=False)
// after
df = df.astype('int64', copy=True)

Defensive patterns

Strategy: validation

Validate before calling

if copy is False:
    copy = True  # Beam cannot honor copy=False

Type guard

def astype_args_are_safe(dtype, copy=True) -> bool:
    return bool(copy) and not (dtype == 'category' and not isinstance(dtype, pd.CategoricalDtype))

Try / catch

try:
    df = df.astype('int64', copy=False)
except frame_base.WontImplementError:
    df = df.astype('int64', copy=True)

Prevention

Never pass copy=False in Beam DataFrame code
Always set copy=True explicitly given pandas default changes across versions
Review perf-only flags when porting pandas code to distributed runtimes

When it happens

Trigger: df.astype(dtype, copy=False) on any DeferredDataFrame/DeferredSeries (copy defaulting to False in some pandas versions also triggers it).

Common situations: Optimization-minded pandas ports that pass copy=False; code written against older pandas where copy defaulted to False; perf-tuned ETL scripts migrated to Beam.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/207286cc90167112. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:637

    semantics.

    ``dtype="category`` is not supported because the type of the output column
    depends on the data. Please use ``pd.CategoricalDtype`` with explicit
    categories instead.
    """
    requires = partitionings.Arbitrary()

    if errors == "ignore":
      # We need all data in order to ignore errors and propagate the original
      # data.
      requires = partitionings.Singleton(
          reason=(
              f"astype(errors={errors!r}) is currently not parallelizable, "
              "because all data must be collected on one node to determine if "
              "the original data should be propagated instead."))

    if not copy:
      raise frame_base.WontImplementError(
          f"astype(copy={copy!r}) is not supported because it relies on "
          "memory-sharing semantics that are not compatible with the Beam "
          "model.")

    # An instance of CategoricalDtype is actualy considered equal to the string
    # 'category', so we have to explicitly check if dtype is an instance of
    # CategoricalDtype, and allow it.
    # See https://github.com/apache/beam/issues/23276
    if dtype == 'category' and not isinstance(dtype, pd.CategoricalDtype):
      raise frame_base.WontImplementError(
          "astype(dtype='category') is not supported because the type of the "
          "output column depends on the data. Please use pd.CategoricalDtype "
          "with explicit categories instead.",
          reason="non-deferred-columns")

    return frame_base.DeferredFrame.wrap(
        expressions.ComputedExpression(
            'astype',

View on GitHub (pinned to 12126d8942)