{"record":{"id":"0fa4f6e35f1270ea","repo":"influxdata/influxdb","slug":"failed-to-allocate-buffer-while-writing-parquet-0","errorCode":null,"errorMessage":"failed to allocate buffer while writing parquet: {0}","messagePattern":"failed to allocate buffer while writing parquet: (.+?)","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"core/parquet_file/src/writer.rs","lineNumber":25,"sourceCode":"    execution::memory_pool::{MemoryConsumer, MemoryPool, MemoryReservation},\n};\nuse parquet::{\n    arrow::ArrowWriter,\n    errors::ParquetError,\n    file::{metadata::ParquetMetaData, properties::WriterProperties},\n};\nuse thiserror::Error;\nuse tracing::warn;\n\n/// Errors related to [`TrackedMemoryArrowWriter`]\n#[derive(Debug, Error)]\npub enum Error {\n    /// Writing the parquet file failed with the specified error.\n    #[error(\"failed to write parquet file: {0}\")]\n    Writer(#[from] ParquetError),\n\n    /// Could not allocate sufficient memory\n    #[error(\"failed to allocate buffer while writing parquet: {0}\")]\n    OutOfMemory(Box<DataFusionError>),\n}\n\n// Manual impl, see https://github.com/dtolnay/thiserror/issues/415\nimpl From<DataFusionError> for Error {\n    fn from(e: DataFusionError) -> Self {\n        Self::OutOfMemory(Box::new(e))\n    }\n}\n\n/// Results!\npub type Result<T, E = Error> = std::result::Result<T, E>;\n\n/// Wraps an [`ArrowWriter`] to track its buffered memory in a\n/// DataFusion [`MemoryPool`]\n///\n/// If the memory used by the `ArrowWriter` exceeds the memory allowed\n/// by the `MemoryPool`, subsequent writes will fail.","sourceCodeStart":7,"sourceCodeEnd":43,"githubUrl":"https://github.com/influxdata/influxdb/blob/06200ef96ba82c5f6727e5038a83af8e722c6875/core/parquet_file/src/writer.rs#L7-L43","documentation":"Error::OutOfMemory signals that the tracked writer could not reserve sufficient tracked memory for a buffer while writing the parquet file. It wraps a DataFusionError (typically ResourcesExhausted) indicating the memory reservation failed.","triggerScenarios":"The memory pool / tracker refuses an allocation request during parquet encoding because the per-query or global memory limit is exhausted.","commonSituations":"Writing very large row groups under a tight memory limit; concurrent writers exhausting the shared pool; misconfigured memory budget smaller than needed row-group size.","solutions":["Increase the memory pool/limit for the write operation","Reduce row group size or writer parallelism to lower peak buffer usage","Retry the write with fewer concurrent writers / batches in flight"],"exampleFix":"// before\nlet pool = MemoryPool::new(64 * 1024 * 1024);\n// after\nlet pool = MemoryPool::new(512 * 1024 * 1024); // sized for row group + parallel writers","handlingStrategy":"retry","validationCode":"// estimate needed memory and compare with pool headroom before writing\nlet est = row_group_count * row_group_size * row_width_estimate;\nif tracker.memory_limit() - tracker.used() < est { return Err(\"insufficient memory headroom\"); }","typeGuard":null,"tryCatchPattern":"match writer.write(batch) {\n    Err(Error::OutOfMemory(df_err)) => {\n        // reduce in-flight batches / row group size, then retry\n        reduce_parallelism();\n        writer.write(batch)?;\n    }\n    Err(e) => return Err(e.into()),\n    Ok(_) => (),\n}","preventionTips":["Size the memory pool for worst-case row group buffer usage","Bound concurrent writer count relative to the pool budget"],"tags":["parquet","memory","allocation"],"backgroundTag":"insufficient-memory","analyzedSha":"06200ef96ba82c5f6727e5038a83af8e722c6875","analyzedAt":"2026-09-19T12:55:30.003Z","contentChangedAt":"2026-09-19T12:55:30.003Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}