quickwit-oss/quickwit · error

failed to send upload split

Error message

failed to send upload split `{:?}`. the publisher is probably dead

What it means

The uploader actor failed to deliver an `UploadSplits` message through the sequencer/publisher channel because the receiving actor's mailbox is closed, meaning the publisher actor has exited. This is a liveness failure inside the indexing pipeline: uploading succeeded but the result could not be forwarded for publication.

Solutions

1. Inspect indexer logs above this message for the root cause of the publisher actor's death (panic, error, supervisor kill) and fix that first.
2. Check memory/resource limits on the indexing node (OOM kills silently stop actors).
3. Ensure the index was not deleted or the pipeline shut down while indexing was in flight.
4. Restart the indexer; the split upload is retried from the source checkpoint on the next pipeline run.

// act Supervisor handles the failure; ensure the pipeline is observed
match uploader_res {
    Err(err) => tracing::error!(%err, "publisher died mid-upload; restart indexer to reprocess checkpoint"),
    Ok(_) => {}
}

When it happens

Trigger: Calling `SplitsUpdateSender::send` when `split_uploaded_tx.send(SequencerCommand::Proceed(...))` returns `Err` because the publisher actor has terminated and its mailbox no longer accepts messages.

Common situations: The publisher actor crashed or was killed (e.g., panic, supervision shutdown, node shutdown) while the uploader was still processing split uploads; pipeline shutdown races during index deletion; OOM kill of the indexer process.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• 1 is always non-zero.
• 60 should be non-zero
• ack_id not found in in-flight
• `append_records` should be called with `position_opt: None`
• `assign_jobs` should return at least one client or fail.

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/0741bcd868f2065f. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-indexing/src/actors/uploader.rs:149

        if let SplitsUpdateSender::Sequencer(split_uploader_tx) = self
            && split_uploader_tx.send(SequencerCommand::Discard).is_err()
        {
            bail!("failed to send cancel command to sequencer: it is probably dead");
        }
        Ok(())
    }

    async fn send(
        self,
        split_update: SplitsUpdate,
        ctx: &ActorContext<Uploader>,
    ) -> anyhow::Result<()> {
        match self {
            SplitsUpdateSender::Sequencer(split_uploaded_tx) => {
                if let Err(publisher_message) =
                    split_uploaded_tx.send(SequencerCommand::Proceed(split_update))
                {
                    bail!(
                        "failed to send upload split `{:?}`. the publisher is probably dead",
                        &publisher_message
                    );
                }
            }
            SplitsUpdateSender::Publisher(publisher_mailbox) => {
                ctx.send_message(&publisher_mailbox, split_update).await?;
            }
        }
        Ok(())
    }
}

#[derive(Clone)]
pub struct Uploader {
    uploader_type: UploaderType,
    metastore: MetastoreServiceClient,
    merge_policy: Arc<dyn MergePolicy>,

View on GitHub (pinned to a39730c5cd)