What causes "backoff never ends" in vectordotdev/vector?

Verify backoff configuration has no finite retry cap that lets next() return None while run expects infinite backoff.

backoff never ends

Error message

backoff never ends

What it means

The aws_s3/sqs source's run loop applies a Backoff to retry failed receive attempts and calls backoff.next().expect("backoff never ends"). Backoff::next by contract never exhausts (it saturates at the max delay), so a None here means the backoff object is in an invalid/already-exhausted state — an internal invariant violation that panics the run task.

Solutions

Verify backoff configuration has no finite retry cap that lets next() return None while run expects infinite backoff.
Replace the expect with a fallback to the max delay: unwrap_or_else(|_| max_delay) so retrying continues.
Fix any code path that resets/replaces self.backoff while the loop is waiting, which could desynchronize its state.
Address the underlying receive failures (credentials, region, network) so backoff never reaches its limit.

Example fix

// before
let delay = self.backoff.next().expect("backoff never ends");
// after
let delay = self.backoff.next().unwrap_or_else(|| self.backoff.max_delay());

Defensive patterns

Strategy: fallback

Try / catch

let delay = self.backoff.next().unwrap_or_else(|| MAX_BACKOFF_DELAY);

Prevention

Use an infinite/saturating backoff implementation when the loop expects endless retries.
Don't reset or reconfigure the backoff concurrently with next().
Fix root-cause receive failures (credentials, region, connectivity) promptly.
Add a test asserting backoff.next() never returns None.

When it happens

Trigger: SQS receive_loop repeatedly failing and advancing the backoff past its terminal state such that next() returns None — only possible if Backoff::next is misused (called after completion) or its max-retry/end configuration allows exhaustion.

Common situations: Prolonged SQS outage/credential failure driving the backoff to its limit; a bug or config change making the backoff finite when run assumes it is infinite.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/8a1c7026c3141290. Report an issue: GitHub.

Appendix: source

Thrown at src/sources/aws_s3/sqs.rs:425

            backoff: ExponentialBackoff::default().max_delay(Duration::from_secs(30)),
        }
    }

    async fn run(mut self) {
        let shutdown = self.shutdown.clone().fuse();
        pin!(shutdown);

        loop {
            select! {
                _ = &mut shutdown => break,
                result = self.run_once() => {
                    match result {
                        Ok(()) => {
                            // Reset backoff on successful receive
                            self.backoff.reset();
                        }
                        Err(_) => {
                            let delay = self.backoff.next().expect("backoff never ends");
                            trace!(
                                delay_ms = delay.as_millis(),
                                "`run_once` failed, will retry after delay.",
                            );
                            tokio::time::sleep(delay).await;
                        }
                    }
                },
            }
        }
    }

    async fn run_once(&mut self) -> Result<(), ()> {
        let messages = match self.receive_messages().await {
            Ok(messages) => {
                emit!(SqsMessageReceiveSucceeded {
                    count: messages.len(),
                });

View on GitHub (pinned to bdb87aeaa4)