{"record":{"id":"d8ea8fa5ce15e7a6","repo":"apache/beam","slug":"astype-dtype-category-is-not-supported-because-the-type-of","errorCode":null,"errorMessage":"astype(dtype='category') is not supported because the type of the output column depends on the data. Please use pd.CategoricalDtype with explicit categories instead.","messagePattern":"astype\\(dtype='category'\\) is not supported because the type of the output column depends on the data\\. Please use pd\\.CategoricalDtype with explicit categories instead\\.","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":647,"sourceCode":"      # data.\n      requires = partitionings.Singleton(\n          reason=(\n              f\"astype(errors={errors!r}) is currently not parallelizable, \"\n              \"because all data must be collected on one node to determine if \"\n              \"the original data should be propagated instead.\"))\n\n    if not copy:\n      raise frame_base.WontImplementError(\n          f\"astype(copy={copy!r}) is not supported because it relies on \"\n          \"memory-sharing semantics that are not compatible with the Beam \"\n          \"model.\")\n\n    # An instance of CategoricalDtype is actualy considered equal to the string\n    # 'category', so we have to explicitly check if dtype is an instance of\n    # CategoricalDtype, and allow it.\n    # See https://github.com/apache/beam/issues/23276\n    if dtype == 'category' and not isinstance(dtype, pd.CategoricalDtype):\n      raise frame_base.WontImplementError(\n          \"astype(dtype='category') is not supported because the type of the \"\n          \"output column depends on the data. Please use pd.CategoricalDtype \"\n          \"with explicit categories instead.\",\n          reason=\"non-deferred-columns\")\n\n    return frame_base.DeferredFrame.wrap(\n        expressions.ComputedExpression(\n            'astype',\n            lambda df: df.astype(dtype=dtype, copy=copy, errors=errors),\n            [self._expr],\n            requires_partition_by=requires,\n            preserves_partition_by=partitionings.Arbitrary()))\n\n  at_time = frame_base._elementwise_method(\n      'at_time', base=pd.core.generic.NDFrame)\n  between_time = frame_base._elementwise_method(\n      'between_time', base=pd.core.generic.NDFrame)\n  copy = frame_base._elementwise_method('copy', base=pd.core.generic.NDFrame)","sourceCodeStart":629,"sourceCodeEnd":665,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L629-L665","documentation":"astype('category') is rejected because the resulting categorical categories are derived from the data itself, which would make the output column types data-dependent — a non-deferred property the Beam DataFrame API forbids (columns must not depend on data values). Passing an explicit pd.CategoricalDtype with fixed categories is allowed.","triggerScenarios":"df.astype('category') or series.astype('category') with the plain string 'category' (and not a pd.CategoricalDtype instance) on a deferred frame.","commonSituations":"Encoding columns to categorical dtype for memory savings or ML preprocessing; pandas ports that relied on implicit category inference; string-to-category conversions in ETL.","solutions":["Use an explicit dtype: df.astype(pd.CategoricalDtype(categories=['a','b','c'])).","Derive the category list from a separate (bounded, collected) pass, then build pd.CategoricalDtype from it.","Keep the categorical conversion in a to_pandas() stage instead.","Use map/replace to encode values to fixed codes rather than categorical dtype."],"exampleFix":"// before\ndf = df.astype('category')\n// after\ndf = df.astype(pd.CategoricalDtype(categories=['low', 'medium', 'high']))","handlingStrategy":"validation","validationCode":"if dtype == 'category':\n    raise ValueError('Use pd.CategoricalDtype(categories=[...]) instead of the string category')","typeGuard":"def is_explicit_categorical(dtype) -> bool:\n    return isinstance(dtype, pd.CategoricalDtype)","tryCatchPattern":"try:\n    df = df.astype('category')\nexcept frame_base.WontImplementError:\n    df = df.astype(pd.CategoricalDtype(categories=['a', 'b', 'c']))","preventionTips":["Never astype to the bare string 'category' in Beam","Define category sets from config or a prior collection pass","Prefer fixed code mappings (map/replace) over inferred categoricals"],"tags":["apache-beam","dataframe","pandas","categorical","astype"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}