vectordotdev/vector · error

Partition consumer finished after completion.

Error message

Partition consumer finished after completion.

What it means

The Kafka source driver moves through ConsumerState::Consuming -> Draining -> Complete. Finished partition tasks arriving via join_next() are routed according to that state; a task finishing when the state machine is already Complete contradicts the design (no tasks should remain), hence unreachable!('Partition consumer finished after completion.')

Source

Thrown at src/sources/kafka.rs:802

    // should completely drain its acknowledgement stream after receiving an end signal
    let mut partition_consumers: JoinSet<(TopicPartition, PartitionConsumerStatus)> =
        Default::default();

    // Handles that will let us end any consumer task that exceeds a drain deadline
    let mut abort_handles: HashMap<TopicPartition, tokio::task::AbortHandle> = HashMap::new();

    let exit_eof = eof.is_some();

    while let ConsumerState::Consuming(_) | ConsumerState::Draining(_) = consumer_state {
        tokio::select! {
            Some(Ok((finished_partition, status))) = partition_consumers.join_next(), if !partition_consumers.is_empty() => {
                debug!("Partition consumer finished for {}:{}", &finished_partition.0, finished_partition.1);
                // If this task ended on its own, the end_signal for it will still be in here.
                end_signals.remove(&finished_partition);
                abort_handles.remove(&finished_partition);

                (drain_deadline, consumer_state) = match consumer_state {
                    ConsumerState::Complete => unreachable!("Partition consumer finished after completion."),
                    ConsumerState::Draining(mut state) => {
                        state.partition_drained(finished_partition);

                        if state.is_drain_complete() {
                            debug!("All expected partitions have drained.");
                            state.finish_drain(drain_deadline)
                        } else {
                            state.keep_draining(drain_deadline)
                        }
                    },
                    ConsumerState::Consuming(state) => {
                        // If we are here, it is likely because the consumer
                        // tasks are set up to exit upon reaching the end of the
                        // partition.
                        if !exit_eof {
                            debug!("Partition consumer task finished, while not in draining mode.");
                        }
                        state.keep_consuming(drain_deadline)

View on GitHub (pinned to 3708c39b12)

Solutions

1. Upgrade Vector - the consumer state machine received fixes around shutdown races
2. Reduce rebalance churn: unique group id per instance, sane session.timeout.ms/heartbeat.interval.ms, stagger rolling restarts
3. Reproduce with debug logging for the kafka module and note the timing between 'Partition consumer finished' and completion
4. Report with logs if it recurs on the latest release

// Invariant violation inside Vector's driver task: contain it at the process level.
// systemd 'Restart=on-failure' + RUST_BACKTRACE=1; in-process supervision:
if let Err(je) = tokio::spawn(run_kafka_source(cfg, shutdown)).await {
    if je.is_panic() { /* emit metric/alert, restart source with backoff */ }
}

When it happens

Trigger: A partition consumer task outliving the driver's transition to Complete - a join_next() result delivered after the state was finalized, or an aborted task's result racing the completion path; a shutdown/rebalance race inside the select! loop.

Common situations: Heavy rebalance churn during shutdown (consumer group membership flapping), ack finalizers delaying task exit past the drain deadline, or rdkafka callback ordering changes across versions; typically intermittent and load-dependent.

Related errors

• Partition assignment received after completion.
• Partitions revoked after completion.
• Shutdown received after completion.
• Drain deadline received after completion.
• MessageStream never calls Ready(None)

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/63a7a53897a842c1. Report an issue: GitHub.