dbt-labs/dbt-core · error

get_csv_data: failed to format CSV: {e}

Error message

get_csv_data: failed to format CSV: {e}

What it means

`get_csv_data` converts the table's underlying Arrow record batch to CSV bytes using `arrow::csv::Writer`. If the Arrow CSV writer fails to serialize the batch (unsupported column type, Arrow writer error, invalid batch state), the error is wrapped as this `InvalidOperation` minijinja error and surfaced to the macro caller.

Source

Thrown at crates/dbt-adapter/src/adapter/mod.rs:4414

        let iter = ArgsIter::new("get_credentials", &["connection_overrides"], args);
        let overrides = iter.next_arg::<Option<&Value>>()?;
        iter.finish()?;
        match &self.inner {
            Typed { adapter, .. } => {
                Ok(adapter.get_credentials(overrides.unwrap_or(&Value::UNDEFINED)))
            }
            Parse(_) => Ok(empty_map_value()),
        }
    }

    pub fn get_csv_data(&self, table: Arc<AgateTable>) -> Result<Value, minijinja::Error> {
        let batch = table.original_record_batch();
        let mut buf: Vec<u8> = Vec::new();
        let mut writer = arrow::csv::WriterBuilder::new()
            .with_header(false)
            .build(&mut buf);
        writer.write(&batch).map_err(|e| {
            minijinja::Error::new(
                minijinja::ErrorKind::InvalidOperation,
                format!("get_csv_data: failed to format CSV: {e}"),
            )
        })?;
        drop(writer);
        Ok(Value::from(String::from_utf8_lossy(&buf).into_owned()))
    }
}

/// Adapter methods whose `Parse`-mode implementation independently
/// fabricates relation/table/column/schema-shaped data (rather than
/// returning a trivial `bool`/`none` placeholder) instead of ever calling
/// `execute`. Each of these needs to be tainted individually at the
/// dispatch point below -- there is no single shared call they all funnel
/// through to taint once. This is the canonical list; minijinja's
/// `INTROSPECTIVE_METHOD_NAMES` (used for the static "does this macro reach
/// an introspective call" analysis) must be kept in sync with it, since
/// minijinja cannot depend on this crate to reuse it directly.

View on GitHub (pinned to 0267ce9170)

Solutions

1. Inspect the inner `{e}` message to identify which column/type the Arrow writer rejected.
2. Flatten or cast unsupported columns (nested lists/structs) to primitive types before calling get_csv_data.
3. Check the seed CSV source for malformed rows that produced an invalid record batch.
4. If the type is legitimately unsupported, fall back to a macro-level CSV rendering of the table's rows/columns.

Example fix

// before (Jinja): table has a nested struct column the Arrow CSV writer cannot serialize
{% set csv = adapter.get_csv_data(agate_table) %}
// after: cast/select only primitive columns first
{% set simple = agate_table.select_columns(['id','name','value']) %}
{% set csv = adapter.get_csv_data(simple) %}

{# Jinja: pre-check column types if the table exposes its schema #}
{% for col in agate_table.column_names %}
  {# cast or drop non-primitive columns before serialization #}
{% endfor %}

{# Jinja: fallback rendering if Arrow CSV serialization fails #}
{% set csv = adapter.get_csv_data(agate_table) %}
{% if csv is undefined %}
  {# build CSV manually from rows as fallback #}
{% endif %}

When it happens

Trigger: Calling `adapter.get_csv_data(agate_table)` where the table's original record batch contains data the Arrow CSV writer cannot serialize — e.g. unusual/nested column types (lists, structs) that the writer rejects, or a corrupt/invalid record batch.

Common situations: Seed files with nested or exotic column types parsed into complex Arrow types; tables whose schema was transformed by custom macros into unsupported types; very wide seeds hitting writer constraints; engine-internal Arrow errors while writing.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• get_csv_data requires agate_table argument
• get_csv_data: argument must be an AgateTable
• {}
• Failed to convert value to JSON: {err}
• Failed to convert value to YAML: {err}

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/913dc381d8befdd7. Report an issue: GitHub.