What causes "[TRANSFORM-AGGREGATOR] Hash table already moved out" in databendlabs/databend?

Upgrade/retry; verify only one finish call and that all input blocks arrive before on_finish

[TRANSFORM-AGGREGATOR] Hash table already moved out

Error message

[TRANSFORM-AGGREGATOR] Hash table already moved out

What it means

TransformPartialAggregate::execute_one_block requires ownership of the partial aggregation hash table to absorb a block. If self.hash_table is HashTable::MovedOut — the table was already moved out (e.g. by on_finish taking it) — the code hits unreachable!, guarding the invariant that no data is processed after the table was handed off.

Solutions

Upgrade/retry; verify only one finish call and that all input blocks arrive before on_finish
If a custom pipeline feeds this transform, ensure blocks stop before finish
File a bug with query profile and stack trace if reproducible

Defensive patterns

Strategy: try-catch

Type guard

fn table_alive(ht: &HashTable) -> bool {
    matches!(ht, HashTable::AggregateHashTable(_))
}

Try / catch

match res {
    Err(e) if e.message().contains("Hash table already moved out") => {
        // query abort; retry on fixed version; report with plan + stack trace
    }
    ...
}

Prevention

Never invoke on_finish while input blocks remain queued for the transform
Add lifecycle assertions in pipeline drivers (no transform after finish)
Keep std::mem::take of hash tables confined to on_finish
Run executor state-machine tests after HashTable enum changes

When it happens

Trigger: transform() delivers another input block after the hash table was moved out via std::mem::take (e.g. on_finish ran before remaining input drained), which can happen with pipeline event-ordering bugs or retry/replay of input blocks.

Common situations: Executor bugs where finish precedes the last transform calls; duplicated blocks in the pipeline driver; regressions from refactoring the HashTable enum.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/c0afc32398c6181a. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/pipelines/processors/transforms/aggregator/transform_aggregate_partial.rs:187

    ) -> Vec<ProjectedBlock<'a>> {
        aggregate_functions_arguments
            .iter()
            .map(|function_arguments| ProjectedBlock::project(function_arguments, block))
            .collect::<Vec<_>>()
    }

    #[inline(always)]
    fn execute_one_block(&mut self, block: DataBlock) -> Result<()> {
        let group_columns = ProjectedBlock::project(&self.params.group_columns, &block);
        let rows_num = block.num_rows();
        let block_bytes = block.memory_size();

        self.statistics.record_block(rows_num, block_bytes);

        {
            match &mut self.hash_table {
                HashTable::MovedOut => {
                    unreachable!("[TRANSFORM-AGGREGATOR] Hash table already moved out")
                }
                HashTable::AggregateHashTable(hashtable) => {
                    let params_columns = Self::aggregate_arguments(
                        &block,
                        &self.params.aggregate_functions_arguments,
                    );
                    let agg_states = (&[]).into();

                    let _ = hashtable.add_groups(
                        &mut self.probe_state,
                        group_columns,
                        &params_columns,
                        agg_states,
                        rows_num,
                    )?;
                    Ok(())
                }
            }

View on GitHub (pinned to 288d84d76e)