{"record":{"id":"62205f81fed3a365","repo":"quickwit-oss/quickwit","slug":"failed-to-send-cancel-command-to-sequencer-it-is","errorCode":null,"errorMessage":"failed to send cancel command to sequencer: it is probably dead","messagePattern":"failed to send cancel command to sequencer: it is probably dead","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-indexing/src/actors/uploader.rs","lineNumber":134,"sourceCode":"            }\n            SplitsUpdateMailbox::Publisher(publisher_mailbox) => {\n                Ok(SplitsUpdateSender::Publisher(publisher_mailbox.clone()))\n            }\n        }\n    }\n}\n\nenum SplitsUpdateSender {\n    Sequencer(Sender<SequencerCommand<SplitsUpdate>>),\n    Publisher(Mailbox<Publisher>),\n}\n\nimpl SplitsUpdateSender {\n    fn discard(self) -> anyhow::Result<()> {\n        if let SplitsUpdateSender::Sequencer(split_uploader_tx) = self\n            && split_uploader_tx.send(SequencerCommand::Discard).is_err()\n        {\n            bail!(\"failed to send cancel command to sequencer: it is probably dead\");\n        }\n        Ok(())\n    }\n\n    async fn send(\n        self,\n        split_update: SplitsUpdate,\n        ctx: &ActorContext<Uploader>,\n    ) -> anyhow::Result<()> {\n        match self {\n            SplitsUpdateSender::Sequencer(split_uploaded_tx) => {\n                if let Err(publisher_message) =\n                    split_uploaded_tx.send(SequencerCommand::Proceed(split_update))\n                {\n                    bail!(\n                        \"failed to send upload split `{:?}`. the publisher is probably dead\",\n                        &publisher_message\n                    );","sourceCodeStart":116,"sourceCodeEnd":152,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-indexing/src/actors/uploader.rs#L116-L152","documentation":"`SplitsUpdateSender::discard` sends a `SequencerCommand::Discard` to the sequencer actor to abort a pending split upload. If the bounded channel send fails, the sequencer actor has already terminated, so the discard cannot be delivered and this error is raised.","triggerScenarios":"Calling `discard` during upload rollback (e.g. after a failed upload or shutdown path) when the Sequencer actor's mailbox is closed because the actor died.","commonSituations":"Sequencer crashed or the indexing pipeline was torn down concurrently; shutdown races where uploader cleanup runs after actor termination.","solutions":["Treat the upload as already abandoned — the sequencer being dead means nothing will be published; check pipeline logs for the sequencer's actual failure.","Rely on the actor framework's supervision/restart to rebuild the pipeline and clean stale state.","If this recurs, investigate why the sequencer exits early (upstream publisher errors, OOM, panic)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// before discard, check the actor handle is still alive if available\nif sequencer_handle.state().is_exit() {\n    // skip discard; the pipeline is already torn down\n    return Ok(());\n}","typeGuard":null,"tryCatchPattern":"if let Err(e) = sender.discard() {\n    if e.to_string().contains(\"probably dead\") {\n        // sequencer already gone: log at debug and treat upload as abandoned\n    } else {\n        return Err(e);\n    }\n}","preventionTips":["Perform discard before or as part of orderly pipeline shutdown, not after actor exit.","Investigate sequencer crash root causes (publisher errors, panics) rather than retrying discard.","Monitor actor restart counters in the indexing pipeline to catch recurring sequencer deaths."],"tags":["actors","sequencer","lifecycle","indexing"],"backgroundTag":"broken-pipe","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}