apache/beam · error · WontImplementError

shift(axis= ) is only supported with freq defined, and…

Error message

shift(axis={axis!r}) is only supported with freq defined, and fill_value undefined (got freq={freq!r},fill_value={fill_value!r}). Other configurations are sensitive to the order of the data because they require populating shifted rows with `fill_value`.

What it means

Beam's DataFrame.shift throws WontImplementError for the row axis unless freq is given and fill_value is left undefined. Without freq, shifting fills vacated positions with fill_value, which depends on positional row order and is not preserved in distributed execution.

Solutions

Provide a freq argument (e.g. freq='D') so the shift is index-based and order-independent.
Remove the fill_value argument; the default fill (via freq shifting) is order-safe.
Implement a windowing/DoFn-based positional shift in core Beam if truly needed.
Fall back to local pandas for positional shifts.

Example fix

// before
shifted = df.shift(1, fill_value=0)
// after
shifted = df.shift(1, freq='D')

Defensive patterns

Strategy: validation

Validate before calling

def check_shift_args(freq, kwargs, axis):
    if axis not in (1, 'columns') and (freq is None or 'fill_value' in kwargs):
        raise ValueError('shift requires freq and no fill_value on the row axis in Beam')

Type guard

def shift_supported(freq, kwargs, axis) -> bool:
    return axis in (1, 'columns') or (freq is not None and 'fill_value' not in kwargs)

Try / catch

from apache_beam.dataframe import frame_base
try:
    shifted = df.shift(1, fill_value=0)
except frame_base.WontImplementError:
    shifted = df.shift(1, freq='D')

Prevention

Shift by time frequency (freq=...), not by position, in Beam pipelines.
Never pass fill_value to shift in Beam code.
Use Beam windowing primitives for order-based time operations.

When it happens

Trigger: Calling df.shift(periods) or df.shift(..., fill_value=...) without freq on a DeferredDataFrame (row axis), or shift with axis='columns' variants lacking freq.

Common situations: Time-series code ported from pandas that shifts by positional periods instead of by time frequency; custom fill values for leading NaNs.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/416b65c5f49463fd. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:3766

    )

  select_dtypes = frame_base._elementwise_method('select_dtypes',
                                                 base=pd.DataFrame)

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  def shift(self, axis, freq, **kwargs):
    """shift with ``axis="index" is only supported with ``freq`` specified and
    ``fill_value`` undefined. Other configurations make this operation
    order-sensitive."""
    if axis in (1, 'columns'):
      preserves = partitionings.Arbitrary()
      proxy = None
    else:
      if freq is None or 'fill_value' in kwargs:
        fill_value = kwargs.get('fill_value', 'NOT SET')
        raise frame_base.WontImplementError(
            f"shift(axis={axis!r}) is only supported with freq defined, and "
            f"fill_value undefined (got freq={freq!r},"
            f"fill_value={fill_value!r}). Other configurations are sensitive "
            "to the order of the data because they require populating shifted "
            "rows with `fill_value`.",
            reason="order-sensitive")
      # proxy generation fails in pandas <1.2
      # Seems due to https://github.com/pandas-dev/pandas/issues/14811,
      # bug with shift on empty indexes.
      # Fortunately the proxy should be identical to the input.
      proxy = self._expr.proxy().copy()


      # index is modified, so no partitioning is preserved.
      preserves = partitionings.Singleton()

    return frame_base.DeferredFrame.wrap(
        expressions.ComputedExpression(

View on GitHub (pinned to 12126d8942)