dbt-labs/dbt-core · error · Exception

{type(df)} is not a supported type for dbt Python materializ

Error message

{type(df)} is not a supported type for dbt Python materialization

What it means

The Apache Spark (dbt-spark) dbt Python table materialization macro checks the `df` object against known DataFrame types — Spark, Databricks Koalas (via to_spark), and pandas (via spark.createDataFrame) — before writing with df.write.saveAsTable. Any other object type raises this Exception because the macro cannot persist it to the target relation.

Source

Thrown at crates/dbt-loader/src/dbt_macro_assets/dbt-spark/macros/materializations/table.sql:99

# since they know how to convert pandas DataFrames better than `spark.createDataFrame(df)`
# and converting from pandas-on-Spark to Spark DataFrame has no overhead
if pyspark_pandas_api_available and pandas_available and isinstance(df, pandas.core.frame.DataFrame):
  df = pyspark.pandas.frame.DataFrame(df)
elif koalas_available and pandas_available and isinstance(df, pandas.core.frame.DataFrame):
  df = databricks.koalas.frame.DataFrame(df)

# convert to pyspark.sql.dataframe.DataFrame
if isinstance(df, pyspark.sql.dataframe.DataFrame):
  pass  # since it is already a Spark DataFrame
elif pyspark_pandas_api_available and isinstance(df, pyspark.pandas.frame.DataFrame):
  df = df.to_spark()
elif koalas_available and isinstance(df, databricks.koalas.frame.DataFrame):
  df = df.to_spark()
elif pandas_available and isinstance(df, pandas.core.frame.DataFrame):
  df = spark.createDataFrame(df)
else:
  msg = f"{type(df)} is not a supported type for dbt Python materialization"
  raise Exception(msg)

df.write.mode("overwrite").format("{{ config.get('file_format', 'delta') }}").option("overwriteSchema", "true").saveAsTable("{{ target_relation }}")
{%- endmacro -%}

{%macro py_script_comment()%}
# how to execute python model in notebook
# dbt = dbtObj(spark.table)
# df = model(dbt, spark)
{%endmacro%}

View on GitHub (pinned to 0267ce9170)

Solutions

1. Ensure the model function returns a pyspark.sql.DataFrame, pandas DataFrame, or databricks.koalas DataFrame.
2. Explicitly convert unsupported types before returning (e.g. `spark.createDataFrame(pandas_df)` or `pdf.to_spark()`).
3. Remove any code that reassigns df to a non-DataFrame (collect()/take()/head() results).
4. Install pyspark/pandas (or databricks.koalas for older runtimes) so the type detection succeeds.

Example fix

# before
def model(dbt, session):
    df = dbt.ref('upstream')
    df = df.collect()  # list
    return df

# after
def model(dbt, session):
    df = dbt.ref('upstream')
    return df  # keep df as a DataFrame

supported = False
try:
    import pyspark
    supported = supported or isinstance(df, pyspark.sql.DataFrame)
except ImportError:
    pass
try:
    import pandas
    supported = supported or isinstance(df, pandas.core.frame.DataFrame)
except ImportError:
    pass
assert supported, f'dbt-spark Python model returned unsupported type {type(df)}'

def is_spark_supported_df(df) -> bool:
    checks = []
    try:
        import pyspark
        checks.append(isinstance(df, pyspark.sql.DataFrame))
    except ImportError:
        pass
    try:
        import pandas
        checks.append(isinstance(df, pandas.core.frame.DataFrame))
    except ImportError:
        pass
    try:
        import databricks.koalas
        checks.append(isinstance(df, databricks.koalas.frame.DataFrame))
    except ImportError:
        pass
    return any(checks)

try:
    dbt_runner.run(model_sql)
except Exception as e:
    if 'is not a supported type for dbt Python materialization' in str(e):
        logger.error('Return a Spark, pandas, or koalas DataFrame from the Python model')
        raise RuntimeError('Unsupported return type in dbt-spark Python model') from e

When it happens

Trigger: A Python model on Spark whose function returns None (no return statement), or returns a type not covered by the isinstance chain: a list/tuple from collect(), a numpy array, a polars DataFrame, a pyspark.pandas DataFrame when neither koalas nor the pandas check matches, or an RDD.

Common situations: Returning df.collect() output instead of the DataFrame; forgetting `return df`; using pyspark.pandas on a runtime where the availability flags don't detect it; accidentally shadowing `df` with a converted value (e.g. df = df.toPandas().values).

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• {type(df)} is not a supported type for dbt Python materializ
• {type(df)} is not a supported type for dbt Python materializ
• {type(df)} is not a supported type for dbt Python materializ
• Schema not found for canonical FQN: {}
• render_for_create is only available for Databricks/Spark

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/c6409901eeba0cce. Report an issue: GitHub.