failed to send cancel command to sequencer: it is probably…

Error message

failed to send cancel command to sequencer: it is probably dead

What it means

`SplitsUpdateSender::discard` sends a `SequencerCommand::Discard` to the sequencer actor to abort a pending split upload. If the bounded channel send fails, the sequencer actor has already terminated, so the discard cannot be delivered and this error is raised.

Solutions

Treat the upload as already abandoned — the sequencer being dead means nothing will be published; check pipeline logs for the sequencer's actual failure.
Rely on the actor framework's supervision/restart to rebuild the pipeline and clean stale state.
If this recurs, investigate why the sequencer exits early (upstream publisher errors, OOM, panic).

Defensive patterns

Strategy: try-catch

Validate before calling

// before discard, check the actor handle is still alive if available
if sequencer_handle.state().is_exit() {
    // skip discard; the pipeline is already torn down
    return Ok(());
}

Try / catch

if let Err(e) = sender.discard() {
    if e.to_string().contains("probably dead") {
        // sequencer already gone: log at debug and treat upload as abandoned
    } else {
        return Err(e);
    }
}

Prevention

Perform discard before or as part of orderly pipeline shutdown, not after actor exit.
Investigate sequencer crash root causes (publisher errors, panics) rather than retrying discard.
Monitor actor restart counters in the indexing pipeline to catch recurring sequencer deaths.

When it happens

Trigger: Calling `discard` during upload rollback (e.g. after a failed upload or shutdown path) when the Sequencer actor's mailbox is closed because the actor died.

Common situations: Sequencer crashed or the indexing pipeline was torn down concurrently; shutdown races where uploader cleanup runs after actor termination.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/62205f81fed3a365. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-indexing/src/actors/uploader.rs:134

            }
            SplitsUpdateMailbox::Publisher(publisher_mailbox) => {
                Ok(SplitsUpdateSender::Publisher(publisher_mailbox.clone()))
            }
        }
    }
}

enum SplitsUpdateSender {
    Sequencer(Sender<SequencerCommand<SplitsUpdate>>),
    Publisher(Mailbox<Publisher>),
}

impl SplitsUpdateSender {
    fn discard(self) -> anyhow::Result<()> {
        if let SplitsUpdateSender::Sequencer(split_uploader_tx) = self
            && split_uploader_tx.send(SequencerCommand::Discard).is_err()
        {
            bail!("failed to send cancel command to sequencer: it is probably dead");
        }
        Ok(())
    }

    async fn send(
        self,
        split_update: SplitsUpdate,
        ctx: &ActorContext<Uploader>,
    ) -> anyhow::Result<()> {
        match self {
            SplitsUpdateSender::Sequencer(split_uploaded_tx) => {
                if let Err(publisher_message) =
                    split_uploaded_tx.send(SequencerCommand::Proceed(split_update))
                {
                    bail!(
                        "failed to send upload split `{:?}`. the publisher is probably dead",
                        &publisher_message
                    );

View on GitHub (pinned to a39730c5cd)