dbt-labs/dbt-core · error

Parquet writer is not initialized

Error message

Parquet writer is not initialized

What it means

Parquet trace writer invariant panic. ParquetWriterLayer guarantees parquet_writer is Some from construction and only takes it during finalize() after a flush, so flush_batch should always see it. Panicking here means the writer was consumed or the layer was used after finalize.

Source

Thrown at crates/dbt-tracing/src/layers/parquet_writer.rs:92

        self.buffer.push(record);

        Ok(())
    }

    fn flush_batch(&mut self) -> TracingResult<()> {
        if self.buffer.is_empty() {
            return Ok(());
        }

        // Serialize records to Arrow RecordBatch
        let record_batch = serialize_to_arrow(&self.buffer, &self.arrow_schemas)
            .map_err(|e| TracingError::io(format!("Failed to serialize to Arrow: {}", e)))?;

        // Write the batch
        let Some(ref mut writer) = self.parquet_writer else {
            // Should not be possible, since we ensure that parquet_writer is Some in new()
            // and we only take it in finalize() after flushing
            unreachable!("Parquet writer is not initialized");
        };

        writer
            .write(&record_batch)
            .map_err(|e| TracingError::io(format!("Failed to write Parquet batch: {}", e)))?;

        // Flush if we are over memory limit
        if writer.memory_size() >= PARQUET_WRITER_MEMORY_LIMIT {
            writer
                .flush()
                .map_err(|e| TracingError::io(format!("Failed to flush Parquet writer: {}", e)))?;
        }

        // Clear buffer for reuse (truncate avoids reallocation)
        self.buffer.truncate(0);

        Ok(())
    }

View on GitHub (pinned to 0267ce9170)

Solutions

1. Ensure finalize() is called exactly once, after all records are written
2. Do not emit tracing events after the parquet layer has been finalized
3. If custom lifecycle code exists, keep parquet_writer Some until the final flush completes
4. Report a bug with a reproducer if triggered through normal usage

Example fix

// before
let Some(ref mut writer) = self.parquet_writer else {
    unreachable!("Parquet writer is not initialized");
};
// after
let Some(ref mut writer) = self.parquet_writer else {
    eprintln!("parquet writer already finalized; dropping batch");
    return Ok(());
};

// check lifecycle before writing
assert!(!self.finalized, "cannot write records after finalize()");

// panics are not recoverable via catch_unwind here; ensure lifecycle:
match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| layer.write_record(rec))) {
    Ok(v) => v,
    Err(_) => eprintln!("trace record dropped: writer finalized"),
}

When it happens

Trigger: write_record or finalize calling flush_batch after finalize() has already taken the parquet_writer, or a construction path that leaves parquet_writer as None.

Common situations: Double-finalize of the tracing subscriber; writing trace records after the tracing pipeline was shut down; reuse of a finalized writer layer.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Parent span must have a SpanStartInfo record in its extensio
• Failed to serialize core event info to JSON
• test tracing subscriber should be valid
• test tracing subscriber should be valid
• attrs_updater should only be called once

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/a2e2f16c4878a1b2. Report an issue: GitHub.