vectordotdev/vector · error

Partitions revoked after completion.

Error message

Partitions revoked after completion.

What it means

Symmetric to assignments: once the driver is Complete no KafkaCallback::PartitionsRevoked can legitimately arrive (callback channels are closed at completion). A revoke in Complete violates the shutdown state machine and panics - the alternative would be silently losing acknowledgement bookkeeping for already-finalized partitions.

Source

Thrown at src/sources/kafka.rs:856

                        let acks = consumer.context().acknowledgements;
                        for tp in assigned_partitions.drain(0..) {
                            let topic = tp.0.as_str();
                            let partition = tp.1;
                            match consumer.split_partition_queue(topic, partition) { Some(pq) => {
                                debug!("Consuming partition {}:{}.", &tp.0, tp.1);
                                let (end_tx, handle) = consumer_state.consume_partition(&mut partition_consumers, tp.clone(), Arc::clone(&consumer), pq, acks, exit_eof);
                                abort_handles.insert(tp.clone(), handle);
                                end_signals.insert(tp, end_tx);
                            } _ => {
                                warn!("Failed to get queue for assigned partition {}:{}.", &tp.0, tp.1);
                            }}
                        }
                        // ensure this is retained until all individual queues are set up
                        drop(done);
                    }
                },
                KafkaCallback::PartitionsRevoked(mut revoked_partitions, drain) => (drain_deadline, consumer_state) = match consumer_state {
                    ConsumerState::Complete => unreachable!("Partitions revoked after completion."),
                    ConsumerState::Draining(d) => {
                        // NB: This would only happen if the task driving the kafka client (i.e. rebalance handlers)
                        // is not handling shutdown signals, and a revoke happens during a shutdown drain; otherwise
                        // this is unreachable code.
                        warn!("Kafka client is already draining revoked partitions.");
                        d.keep_draining(drain_deadline)
                    },
                    ConsumerState::Consuming(state) => {
                        let (deadline, mut state) = state.begin_drain(max_drain_ms, drain, false);

                        for tp in revoked_partitions.drain(0..) {
                            match end_signals.remove(&tp) { Some(end) => {
                                debug!("Revoking partition {}:{}", &tp.0, tp.1);
                                state.revoke_partition(tp, end);
                            } _ => {
                                debug!("Consumer task for partition {}:{} already finished.", &tp.0, tp.1);
                            }}
                        }

View on GitHub (pinned to 3708c39b12)

Solutions

1. Upgrade Vector to pick up state machine race fixes
2. Minimize rebalances during shutdown: quiesce producers before restarting Vector, stagger restarts, prefer cooperative-sticky assignors
3. Collect debug logs showing the revoke-after-complete sequence and open an issue
4. As a workaround in flaky environments, run a single consumer per group so external rebalances cannot interleave

// Same containment shape: the driver task panic is fatal for the source,
// so supervise and recover at the process/task boundary:
if let Err(je) = tokio::spawn(kafka::run(cfg, shutdown)).await {
    if je.is_panic() { /* log, alert, restart with exponential backoff */ }
}

When it happens

Trigger: A revoke callback delivered after the state machine completed - shutdown drain finishing while a group rebalance or broker-initiated revoke is still in flight on the rdkafka callback thread; aborted-task cleanup racing the completion transition.

Common situations: Rolling restarts or k8s evictions during active rebalances; consumers sharing the group with different session timeouts; rdkafka version drift changing callback ordering.

Related errors

• Partition assignment received after completion.
• Partition consumer finished after completion.
• Shutdown received after completion.
• MessageStream never calls Ready(None)
• Drain deadline received after completion.

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/c19c7491073b6ffc. Report an issue: GitHub.