What causes "async crash me panic" in databendlabs/databend?

Ensure the pool outlives all workers: shut down spill workers before closing the pool.

async crash me panic

Error message

async crash me panic

What it means

alloc_buffer waits on a channel of pre-allocated buffers from the spill memory pool. If the channel's sender is dropped (the pool has been shut down) recv_blocking fails and this io::Error (BrokenPipe) is returned instead of a buffer. It signals use of the spill buffer pool after its closure, not transient backpressure.

Solutions

Ensure the pool outlives all workers: shut down spill workers before closing the pool.
Treat the error as terminal — stop the spill operation rather than retrying allocation.
Guard the allocation path with a pool-closed state check before calling alloc_buffer.
If seen during normal (non-shutdown) operation, audit the pool's drop/close path for premature closure.

Example fix

// before
let buf = pool.alloc_buffer().await?;
// after
if pool.is_closed() { return Ok(None); } // graceful stop on shutdown
let buf = pool.alloc_buffer().await.map_err(|e| if is_pool_closed(&e) { SpillError::Stopped } else { e.into() })?;

Defensive patterns

Strategy: try-catch

Validate before calling

if pool_closed.load(Ordering::Acquire) { return Err(SpillError::PoolClosed); }

Type guard

fn pool_alive(pool: &AsyncBuffer) -> bool { !pool.is_closed() }

Try / catch

match alloc_buffer(&pool).await { Err(e) if e.kind() == io::ErrorKind::BrokenPipe && pool.is_closed() => Ok(CancelSpill), Err(e) => Err(e.into()), Ok(buf) => Ok(Use(buf)) }

Prevention

Join/close all spill workers before dropping the pool
Expose and check an is_closed flag before allocating
Never retry allocation after this error — it is terminal
Cover shutdown-during-spill in integration tests

When it happens

Trigger: Calling alloc_buffer on an AsyncBuffer/spill pool that has already been closed; a race where the pool is dropped while a worker is still allocating.

Common situations: Query cancellation/shutdown tearing down the memory pool while spill workers are mid-flight; operator restart logic attempting to allocate from a dead pool; tests dropping the pool early.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/81cc8fb2cdeb1901. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/table_functions/async_crash_me.rs:155

    pub fn create(
        ctx: Arc<dyn TableContext>,
        output: Arc<OutputPort>,
        message: Option<String>,
    ) -> Result<ProcessorPtr> {
        AsyncSourcer::create(ctx.get_scan_progress(), output, AsyncCrashMeSource {
            message,
        })
    }
}

#[async_trait::async_trait]
impl AsyncSource for AsyncCrashMeSource {
    const NAME: &'static str = "async_crash_me";

    #[async_backtrace::framed]
    async fn generate(&mut self) -> Result<Option<DataBlock>> {
        match &self.message {
            None => panic!("async crash me panic"),
            Some(message) => panic!("{}", message),
        }
    }
}

impl TableFunction for AsyncCrashMeTable {
    fn function_name(&self) -> &str {
        self.name()
    }

    fn as_table<'a>(self: Arc<Self>) -> Arc<dyn Table + 'a>
    where Self: 'a {
        self
    }
}

View on GitHub (pinned to 288d84d76e)