influxdata/influxdb · error · Error

failed to allocate buffer while writing parquet

Error message

failed to allocate buffer while writing parquet: {0}

What it means

Error::OutOfMemory signals that the tracked writer could not reserve sufficient tracked memory for a buffer while writing the parquet file. It wraps a DataFusionError (typically ResourcesExhausted) indicating the memory reservation failed.

Solutions

1. Increase the memory pool/limit for the write operation
2. Reduce row group size or writer parallelism to lower peak buffer usage
3. Retry the write with fewer concurrent writers / batches in flight

Example fix

// before
let pool = MemoryPool::new(64 * 1024 * 1024);
// after
let pool = MemoryPool::new(512 * 1024 * 1024); // sized for row group + parallel writers

// estimate needed memory and compare with pool headroom before writing
let est = row_group_count * row_group_size * row_width_estimate;
if tracker.memory_limit() - tracker.used() < est { return Err("insufficient memory headroom"); }

match writer.write(batch) {
    Err(Error::OutOfMemory(df_err)) => {
        // reduce in-flight batches / row group size, then retry
        reduce_parallelism();
        writer.write(batch)?;
    }
    Err(e) => return Err(e.into()),
    Ok(_) => (),
}

When it happens

Trigger: The memory pool / tracker refuses an allocation request during parquet encoding because the per-query or global memory limit is exhausted.

Common situations: Writing very large row groups under a tight memory limit; concurrent writers exhausting the shared pool; misconfigured memory budget smaller than needed row-group size.

Related errors

• failed to build parquet file
• cannot write parquet to a terminal, use `--output
• Could not convert Parquet schema to Arrow schema
• Could not find a time column
• Could not find time column statistics

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/0fa4f6e35f1270ea. Report an issue: GitHub.

Appendix: source

Thrown at core/parquet_file/src/writer.rs:25

    execution::memory_pool::{MemoryConsumer, MemoryPool, MemoryReservation},
};
use parquet::{
    arrow::ArrowWriter,
    errors::ParquetError,
    file::{metadata::ParquetMetaData, properties::WriterProperties},
};
use thiserror::Error;
use tracing::warn;

/// Errors related to [`TrackedMemoryArrowWriter`]
#[derive(Debug, Error)]
pub enum Error {
    /// Writing the parquet file failed with the specified error.
    #[error("failed to write parquet file: {0}")]
    Writer(#[from] ParquetError),

    /// Could not allocate sufficient memory
    #[error("failed to allocate buffer while writing parquet: {0}")]
    OutOfMemory(Box<DataFusionError>),
}

// Manual impl, see https://github.com/dtolnay/thiserror/issues/415
impl From<DataFusionError> for Error {
    fn from(e: DataFusionError) -> Self {
        Self::OutOfMemory(Box::new(e))
    }
}

/// Results!
pub type Result<T, E = Error> = std::result::Result<T, E>;

/// Wraps an [`ArrowWriter`] to track its buffered memory in a
/// DataFusion [`MemoryPool`]
///
/// If the memory used by the `ArrowWriter` exceeds the memory allowed
/// by the `MemoryPool`, subsequent writes will fail.

View on GitHub (pinned to 06200ef96b)