databendlabs/databend · error

Buffer pool working queue need unbounded.

Error message

Buffer pool working queue need unbounded.

What it means

AsyncBuffer's memory-pool `operator()` submits work to a crossbeam working_queue via try_send and expects the queue to be unbounded, so try_send can never fail with Full/Disconnected. The panic fires when the channel is bounded and full, or has been disconnected (receiver dropped) — a queue configuration or lifecycle violation.

Solutions

1. Check logs for a prior panic in the buffer-pool worker thread — restart the query/node to recreate the worker.
2. Verify the working_queue is created with unbounded capacity; fix the constructor if bounded.
3. Reduce concurrent spill/fetch load or increase pool workers so the queue drains.
4. Patch operator() to use send().await or map try_send errors instead of expect.

Example fix

// before
self.working_queue.try_send(op).expect("Buffer pool working queue need unbounded.");
// after
if let Err(e) = self.working_queue.try_send(op) {
    log::error!("buffer pool queue rejected op: {e}");
}

// verify worker is alive and queue unbounded
assert!(buffer_worker_is_running(), "buffer pool worker died; recreate pool");

fn queue_accepts(q: &crossbeam::channel::Sender<BufferOperator>) -> bool { !q.is_full() && !q.is_disconnected() }

match self.working_queue.try_send(op) {
    Err(crossbeam::channel::TrySendError::Disconnected(_)) => recreate_worker_and_retry(op),
    Err(e) => log::error!("queue send failed: {e}"),
    Ok(()) => {}
}

When it happens

Trigger: Working queue constructed bounded (capacity set) and filled faster than the worker drains it; worker thread panicked/exited so the channel is disconnected while fetch_ranges still calls operator().

Common situations: Spill-heavy workloads flooding the buffer pool queue; memory pressure killing the background worker; misconfigured pool size where the worker can't keep up.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• internal error: entered unreachable code
• async crash me panic
• {}
• Temp table id used up
• Invalid temp table desc

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/3b521e67a339682a. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/spillers/async_buffer.rs:200

                    let mut background = Background::create();
                    while let Ok(op) = working_queue.recv().await {
                        let span = Span::enter_with_parent("Background::recv", op.span());
                        background.recv(op).in_span(span).await;
                    }
                }),
            );
        }

        Ok(Arc::new(SpillsBufferPool {
            _runtime: runtime,
            working_queue: working_tx,
        }))
    }

    pub(crate) fn operator(&self, op: BufferOperator) {
        self.working_queue
            .try_send(op)
            .expect("Buffer pool working queue need unbounded.");
    }

    pub fn buffer_write(self: &Arc<Self>, writer: Writer, pool_bytes: usize) -> BufferWriter {
        let (buffer_tx, buffer_rx) = async_channel::unbounded::<Bytes>();
        let memory_pool = MemoryPool::create(pool_bytes);

        let response = BufferOperatorResp::pending();

        self.operator(BufferOperator::WriterTask(BufferWriterTaskOperator {
            writer,
            buffer_rx,
            response: response.clone(),
            memory_pool: memory_pool.clone(),
            span: Span::enter_with_local_parent("BufferWriterTask"),
        }));

        BufferWriter {
            buffer_tx,

View on GitHub (pinned to 288d84d76e)