influxdata/influxdb · error · Error

failed to write parquet file

Error message

failed to write parquet file: {0}

What it means

Error::Writer wraps an underlying arrow/parquet ParquetError produced by TrackedMemoryArrowWriter while serializing record batches to the parquet format. The library surfaces the raw ParquetError message prefixed with 'failed to write parquet file'.

Solutions

  1. Inspect the inner ParquetError message for the root cause and fix accordingly
  2. Ensure every batch written matches the schema the writer was created with
  3. Check that the underlying sink (memory buffer/object store) is writable and has capacity

Example fix

// before
let writer = TrackedMemoryArrowWriter::new(sink, schema_a);
writer.write(batch_with_schema_b)?;
// after
assert_eq!(batch.schema(), schema_a, "batch schema must match writer schema");
writer.write(batch)?;
Defensive patterns

Strategy: try-catch

Validate before calling

assert_eq!(batch.schema().as_ref(), writer_schema.as_ref(), "batch schema must match writer schema");

Try / catch

match writer.write(batch) {
    Err(e @ Error::Writer(pe)) => log::error!("parquet write failed: {pe}"),
    Err(e) => return Err(e.into()),
    Ok(_) => (),
}

Prevention

When it happens

Trigger: Writing record batches fails inside the tracked writer: unsupported data types for parquet, Arrow schema incompatible with the writer's schema, or an I/O failure on the underlying sink.

Common situations: Mismatch between the schema the writer was created with and the batches actually written; attempting to write dictionary/nested types unsupported by the chosen parquet version; disk full or object-store I/O errors.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/66af80fb019065f4. Report an issue: GitHub.

Appendix: source

Thrown at core/parquet_file/src/writer.rs:21

use arrow::{datatypes::SchemaRef, record_batch::RecordBatch};
use datafusion::{
    error::DataFusionError,
    execution::memory_pool::{MemoryConsumer, MemoryPool, MemoryReservation},
};
use parquet::{
    arrow::ArrowWriter,
    errors::ParquetError,
    file::{metadata::ParquetMetaData, properties::WriterProperties},
};
use thiserror::Error;
use tracing::warn;

/// Errors related to [`TrackedMemoryArrowWriter`]
#[derive(Debug, Error)]
pub enum Error {
    /// Writing the parquet file failed with the specified error.
    #[error("failed to write parquet file: {0}")]
    Writer(#[from] ParquetError),

    /// Could not allocate sufficient memory
    #[error("failed to allocate buffer while writing parquet: {0}")]
    OutOfMemory(Box<DataFusionError>),
}

// Manual impl, see https://github.com/dtolnay/thiserror/issues/415
impl From<DataFusionError> for Error {
    fn from(e: DataFusionError) -> Self {
        Self::OutOfMemory(Box::new(e))
    }
}

/// Results!
pub type Result<T, E = Error> = std::result::Result<T, E>;

/// Wraps an [`ArrowWriter`] to track its buffered memory in a

View on GitHub (pinned to 06200ef96b)