dbt-labs/dbt-core · error · Exception

{type(df)} is not a supported type for dbt Python materializ

Error message

{type(df)} is not a supported type for dbt Python materialization

What it means

Raised by the dbt-databricks Python model adapter macro when the DataFrame returned by a Python model is not a Spark DataFrame, a Koalas DataFrame, or a pandas DataFrame convertible via spark.createDataFrame. Only these three types can be written back to the table with the subsequent df.write call.

Source

Thrown at crates/dbt-loader/src/dbt_macro_assets/dbt-databricks/macros/adapters/python.sql:53

            else:
                raise e
    elif koalas_available:
        df = databricks.koalas.frame.DataFrame(df)

# convert to pyspark.sql.dataframe.DataFrame
if isinstance(df, pyspark.sql.dataframe.DataFrame):
    pass  # since it is already a Spark DataFrame
elif newer_pyspark_available and isinstance(df, pyspark.sql.connect.dataframe.DataFrame):
    pass  # since it is already a Spark DataFrame
elif pyspark_pandas_api_available and isinstance(df, pyspark.pandas.frame.DataFrame):
    df = df.to_spark()
elif koalas_available and isinstance(df, databricks.koalas.frame.DataFrame):
    df = df.to_spark()
elif pandas_available and isinstance(df, pandas.core.frame.DataFrame):
    df = spark.createDataFrame(df)
else:
    msg = f"{type(df)} is not a supported type for dbt Python materialization"
    raise Exception(msg)

writer = (
    df.write
        .mode("overwrite")
        .option("overwriteSchema", "true")
{{ py_get_writer_options()|indent(8, True) }}
)

writer.saveAsTable("{{ target_relation }}")
{% endmacro %}

# Note: this is not the code used for performing incremental merges.
# The current process uses this code to create a staging table that is
# merged in using a SQL statement.  To see your incremental config in action,
# look in the dbt.log

{%- macro py_get_writer_options() -%}
{%- set location_root = config.get('location_root', validator=validation.any[basestring]) -%}

View on GitHub (pinned to 0267ce9170)

Solutions

  1. Return a pyspark.sql.DataFrame from the model (e.g. df = spark.createDataFrame(...)) so no conversion is needed.
  2. Convert other DataFrame types to pandas before returning (polars: df.to_pandas()), letting spark.createDataFrame handle it.
  3. Ensure pandas is available on the Databricks cluster runtime used for Python models so the pandas branch works.
  4. Verify the model function returns a DataFrame and not a list/dict/NumPy array.
  5. On newer runtimes, return a pyspark DataFrame rather than relying on the deprecated databricks.koalas branch.

Example fix

# before
def model(dbt, session):
    return pl.from_pandas(pandas_df)  # polars -> exception
# after
def model(dbt, session):
    return pandas_df  # pandas is converted via spark.createDataFrame
Defensive patterns

Strategy: type-guard

Validate before calling

def ensure_supported_df(df):
    from pyspark.sql import DataFrame as SparkDF
    if isinstance(df, SparkDF):
        return df
    import pandas as pd
    if isinstance(df, pd.DataFrame):
        return df  # converted by macro via spark.createDataFrame
    if hasattr(df, 'to_pandas'):
        return df.to_pandas()
    raise TypeError(f'{type(df)} unsupported; return Spark/Koalas/pandas DataFrame')

Type guard

def is_supported_df(df) -> bool:
    try:
        from pyspark.sql import DataFrame as SparkDF
        if isinstance(df, SparkDF):
            return True
    except ImportError:
        pass
    try:
        import pandas as pd
        if isinstance(df, pd.DataFrame):
            return True
    except ImportError:
        pass
    try:
        import databricks.koalas
        if isinstance(df, databricks.koalas.frame.DataFrame):
            return True
    except ImportError:
        pass
    return False

Try / catch

try:
    result = runner.invoke(['run', '--select', 'my_python_model'])
except Exception as e:
    if 'is not a supported type for dbt Python materialization' in str(e):
        fix_model_return_type('my_python_model')
    else:
        raise

Prevention

When it happens

Trigger: A Databricks Python model returns a type other than pyspark.sql.DataFrame, databricks.koalas DataFrame, or pandas.DataFrame — e.g. a polars DataFrame, a list, a NumPy array, or a pyspark.pandas DataFrame (which this macro does not check). Also occurs when the pandas/koalas availability checks are False in the cluster environment despite the model returning pandas.

Common situations: Returning polars or other modern DataFrame libraries from a Python model; returning a plain Python collection instead of a DataFrame; missing pandas in the cluster's dbt environment so pandas_available is False; copy-pasting model code that returns Koalas when Koalas is unavailable on the runtime.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/45b6f39e39a0921a. Report an issue: GitHub.