vectordotdev/vector · error

Drain deadline received after completion.

Error message

Drain deadline received after completion.

What it means

Drain deadlines (the max_drain_ms acknowledgement drain timeout) are armed when partitions are revoked/shutting down and fire in Draining state. If the deadline future wakes while the state is already Complete - deadlines should have been consumed or dropped at completion - the invariant is broken and it panics.

Source

Thrown at src/sources/kafka.rs:923

                                    state.revoke_partition(tp, end);
                                } _ => {
                                    debug!("Consumer task for partition {}:{} already finished.", &tp.0, tp.1);
                                }}
                            });
                        }
                        // If shutdown was initiated by partition EOF mode, the drain phase
                        // will already be complete and would time out if not accounted for here
                        if state.is_drain_complete() {
                            state.finish_drain(deadline)
                        } else {
                            state.keep_draining(deadline)
                        }
                    }
                },
            },

            Some(_) = &mut drain_deadline => (drain_deadline, consumer_state) = match consumer_state {
                ConsumerState::Complete => unreachable!("Drain deadline received after completion."),
                ConsumerState::Consuming(state) => {
                    warn!("A drain deadline fired outside of draining mode.");
                    state.keep_consuming(None.into())
                },
                ConsumerState::Draining(mut draining) => {
                    debug!("Acknowledgement drain deadline reached. Dropping any pending ack streams for revoked partitions.");
                    for tp in draining.consumer_state.expect_drain.drain() {
                        if let Some(handle) = abort_handles.remove(&tp) {
                            handle.abort();
                        }
                    }
                    draining.finish_drain(drain_deadline)
                }
            },
        }
    }
}

View on GitHub (pinned to 3708c39b12)

Solutions

1. Upgrade Vector - timer/branch handling around drain completion was fixed over releases
2. Give max_drain_ms comfortable margin above worst-case acknowledgement latency so the deadline rarely fires at the boundary
3. Capture debug logs ('Acknowledgement drain deadline reached' near completion) and report
4. If embedding, prefer graceful shutdown that awaits the source future instead of cancelling it mid-drain

// Timer/state race inside Vector: contain and restart the source task:
if let Err(je) = tokio::spawn(kafka::run(cfg, shutdown)).await {
    if je.is_panic() { /* log, restart with backoff; attach debug logs to a report */ }
}

When it happens

Trigger: The drain-deadline sleep resolving after the transition to Complete (timer not cancelled on the completion path), or a duplicated deadline future; both are shutdown-race shapes inside the driver's select! loop.

Common situations: Very short max_drain_ms combined with fast drains, tokio timer pressure under load, and versions where the deadline wasn't taken/dropped on the completion branch.

Related errors

• Partition consumer finished after completion.
• Shutdown received after completion.
• Partition assignment received after completion.
• Partitions revoked after completion.
• ServiceSink service sender dropped.

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/8858a533c8a2cfe3. Report an issue: GitHub.