{"record":{"id":"b03b8119eb526c86","repo":"quickwit-oss/quickwit","slug":"fatal-error-non-retryable-batch-error-propagated","errorCode":null,"errorMessage":"fatal_error (non-retryable batch error propagated via bail!)","messagePattern":"fatal_error \\(non-retryable batch error propagated via bail!\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-indexing/src/source/queue_sources/sqs_queue.rs","lineNumber":158,"sourceCode":"                    .set_entries(Some(batch.clone()))\n                    .send()\n            })\n            .await;\n            match res {\n                Ok(res) => {\n                    message_errors.extend(res.failed);\n                }\n                Err(err) => {\n                    batch_errors.push(err);\n                }\n            }\n        }\n        if batch_errors.iter().any(|err| !err.is_retryable()) {\n            let fatal_error = batch_errors\n                .into_iter()\n                .find(|err| !err.is_retryable())\n                .unwrap();\n            bail!(fatal_error);\n        } else if !batch_errors.is_empty() {\n            rate_limited_error!(\n                limit_per_min = 10,\n                count = batch_errors.len(),\n                first_err = ?batch_errors.into_iter().next().unwrap(),\n                \"failed to acknowledge some message batches\",\n            );\n        }\n        // The documentation is unclear about these partial failures. We assume\n        // it is either:\n        // - a transient failure\n        // - the message is already acknowledged\n        // - the message is expired\n        if !message_errors.is_empty() {\n            rate_limited_error!(\n                limit_per_min = 10,\n                count = message_errors.len(),\n                first_err = ?message_errors.into_iter().next().unwrap(),","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-indexing/src/source/queue_sources/sqs_queue.rs#L140-L176","documentation":"SQS queue source acknowledge() batches delete-message calls; if any returned batch error is non-retryable, it is fatal and propagated via bail! rather than being retried or logged away. Retryable batch errors are only rate-limited logged and left for a later attempt.","triggerScenarios":"Calling acknowledge when the AWS SQS DeleteMessageBatch response contains at least one error whose is_retryable() is false (e.g., invalid message handle — message already deleted or visibility expired and deleted, or invalid permission).","commonSituations":"Messages whose visibility timeout expired and were re-delivered/deleted so the receipt handle is stale; IAM policy missing sqs:DeleteMessage; long processing without deadline renewal causing invalid handles.","solutions":["Ensure message deadlines are renewed (modify_deadlines) so receipt handles stay valid before acknowledgement.","Check IAM permissions for sqs:DeleteMessage on the queue.","Treat the fatal error upstream: the source will restart; verify whether the affected messages were already processed to avoid duplicates."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Retryable errors are logged and retried; non-retryable ones abort the actor.\n// Callers should let the pipeline restart and rely on at-least-once redelivery.\nif let Err(fatal) = queue_source.acknowledge(&ack_ids).await {\n    log::error!(\"fatal acknowledge error, source will restart: {fatal}\");\n}","preventionTips":["Renew message deadlines so receipt handles never go stale.","Grant sqs:DeleteMessage (and sqs:ReceiveMessage) to the Quickwit IAM role.","Process messages faster than the visibility timeout or extend it."],"tags":["sqs","aws","acknowledgement","batch-error"],"backgroundTag":"api-error-response","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}