{"record":{"id":"8a1c7026c3141290","repo":"vectordotdev/vector","slug":"backoff-never-ends","errorCode":null,"errorMessage":"backoff never ends","messagePattern":"backoff never ends","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/sources/aws_s3/sqs.rs","lineNumber":425,"sourceCode":"            backoff: ExponentialBackoff::default().max_delay(Duration::from_secs(30)),\n        }\n    }\n\n    async fn run(mut self) {\n        let shutdown = self.shutdown.clone().fuse();\n        pin!(shutdown);\n\n        loop {\n            select! {\n                _ = &mut shutdown => break,\n                result = self.run_once() => {\n                    match result {\n                        Ok(()) => {\n                            // Reset backoff on successful receive\n                            self.backoff.reset();\n                        }\n                        Err(_) => {\n                            let delay = self.backoff.next().expect(\"backoff never ends\");\n                            trace!(\n                                delay_ms = delay.as_millis(),\n                                \"`run_once` failed, will retry after delay.\",\n                            );\n                            tokio::time::sleep(delay).await;\n                        }\n                    }\n                },\n            }\n        }\n    }\n\n    async fn run_once(&mut self) -> Result<(), ()> {\n        let messages = match self.receive_messages().await {\n            Ok(messages) => {\n                emit!(SqsMessageReceiveSucceeded {\n                    count: messages.len(),\n                });","sourceCodeStart":407,"sourceCodeEnd":443,"githubUrl":"https://github.com/vectordotdev/vector/blob/bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013/src/sources/aws_s3/sqs.rs#L407-L443","documentation":"The aws_s3/sqs source's run loop applies a Backoff to retry failed receive attempts and calls backoff.next().expect(\"backoff never ends\"). Backoff::next by contract never exhausts (it saturates at the max delay), so a None here means the backoff object is in an invalid/already-exhausted state — an internal invariant violation that panics the run task.","triggerScenarios":"SQS receive_loop repeatedly failing and advancing the backoff past its terminal state such that next() returns None — only possible if Backoff::next is misused (called after completion) or its max-retry/end configuration allows exhaustion.","commonSituations":"Prolonged SQS outage/credential failure driving the backoff to its limit; a bug or config change making the backoff finite when run assumes it is infinite.","solutions":["Verify backoff configuration has no finite retry cap that lets next() return None while run expects infinite backoff.","Replace the expect with a fallback to the max delay: unwrap_or_else(|_| max_delay) so retrying continues.","Fix any code path that resets/replaces self.backoff while the loop is waiting, which could desynchronize its state.","Address the underlying receive failures (credentials, region, network) so backoff never reaches its limit."],"exampleFix":"// before\nlet delay = self.backoff.next().expect(\"backoff never ends\");\n// after\nlet delay = self.backoff.next().unwrap_or_else(|| self.backoff.max_delay());","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"let delay = self.backoff.next().unwrap_or_else(|| MAX_BACKOFF_DELAY);","preventionTips":["Use an infinite/saturating backoff implementation when the loop expects endless retries.","Don't reset or reconfigure the backoff concurrently with next().","Fix root-cause receive failures (credentials, region, connectivity) promptly.","Add a test asserting backoff.next() never returns None."],"tags":["rust","panic","backoff","retry","aws","sqs"],"backgroundTag":"internal-invariant-violation","analyzedSha":"bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013","analyzedAt":"2026-09-16T02:53:35.741Z","contentChangedAt":"2026-09-16T02:53:35.741Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}