[TRANSFORM-AGGREGATOR] Invalid hash table state during…

Error message

[TRANSFORM-AGGREGATOR] Invalid hash table state during spill check

What it means

TransformFinalAggregate::spill_out spills per-bucket payloads and expects self.hashtable to be HashTable::AggregateHashTable; if it has been moved out (HashTable::MovedOut) or is otherwise in an invalid state, unreachable! aborts. It guards the invariant that a spill check can only run while the final aggregation hash table is still owned by the transform.

Solutions

Upgrade to a version with the transform state-machine fix and retry the query
Check for duplicate finish/spill invocation in any custom pipeline wiring
File a bug with the stack trace and query profile if reproducible

Defensive patterns

Strategy: try-catch

Type guard

fn hashtable_present(ht: &HashTable) -> Option<&AggregateHashTable> {
    match ht {
        HashTable::AggregateHashTable(t) => Some(t),
        HashTable::MovedOut => None,
    }
}

Try / catch

match res {
    Err(e) if e.message().contains("Invalid hash table state during spill check") => {
        // retry query; reduce memory pressure or disable spill path; report if reproducible
    }
    ...
}

Prevention

Avoid custom pipeline wiring that can call finish before spill checks
Keep executor lifecycle tests for spill + finish interleavings
Upgrade promptly; such state-machine bugs are fixed in patch releases
Watch memory/spill settings so spill paths execute under normal conditions

When it happens

Trigger: check_spill -> spill_out runs after the hash table was already taken (e.g. via std::mem::take in on_finish or MovedOut marking), so the else branch fires — typically from double-finish calls, pipeline event ordering bugs, or interrupted/retried pipeline execution.

Common situations: Query cancellation racing with spill; executor bugs invoking finish and spill paths out of order; regressions after refactoring the HashTable enum state machine.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/8f60d5d5f5a69cf7. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/pipelines/processors/transforms/aggregator/transform_aggregate_final.rs:314

                        check_interrupt()?;
                        self.handle_meta(AggregateMeta::from(item), need_check_spill)?;
                    }
                }
            },
        }
        Ok(())
    }

    fn spill_out(&mut self) -> Result<()> {
        self.spilled_occurred = true;
        if let HashTable::AggregateHashTable(v) = mem::take(&mut self.hashtable) {
            for (bucket, payload) in v.payload.into_non_empty_bucket_payloads() {
                check_interrupt()?;
                let data_block = payload.aggregate_flush_all()?.consume_convert_to_full();
                self.spiller.spill(bucket, data_block)?;
            }
        } else {
            unreachable!("[TRANSFORM-AGGREGATOR] Invalid hash table state during spill check")
        }
        self.reset_hashtable(self.current_partition_depth);
        Ok(())
    }

    fn finish(
        &mut self,
        task_id: Option<u64>,
        spilled_depth: usize,
        tx: Sender<FinalAggregateTask>,
    ) -> Result<()> {
        if self.spilled_occurred {
            let (output_rows, hash_index_resizes) = match &self.hashtable {
                HashTable::AggregateHashTable(ht) => {
                    (ht.payload.len(), ht.hash_index_resize_count())
                }
                _ => unreachable!("[TRANSFORM-AGGREGATOR] Invalid hash table state before spill"),
            };

View on GitHub (pinned to 288d84d76e)