xai-org/grok-build · error · anyhow::Error

worker dropped completion channel

Error message

worker dropped completion channel

What it means

Raised when awaiting the oneshot completion channel (`rx.await`) for an enqueued upload and the sender was dropped before replying — meaning the queue worker panicked, was cancelled, or shut down without completing the item. The QueueClosed error is wrapped with 'worker dropped completion channel' to distinguish worker death from an explicit queue-close rejection.

Source

Thrown at crates/codegen/xai-file-utils/src/queue.rs:992

                    remove_item_files(&rejected, Some(&self.stats));
                    return Err(anyhow::Error::new(QueueClosed).context("upload queue closed"));
                }
                if let Some(sidecar) = &rejected.sidecar_path {
                    try_remove_temp(sidecar, Some(&self.stats));
                }
                self.stats.enqueue_fallbacks.fetch_add(1, Ordering::Relaxed);
                self.spawn_inline_upload_owned_snapshot(
                    rejected.source.path().to_path_buf(),
                    gcs_path.to_string(),
                    content_type.to_string(),
                    size,
                    rejected.completion_tx,
                );
            }
        }
        rx.await
            .map_err(|_| {
                anyhow::Error::new(QueueClosed).context("worker dropped completion channel")
            })?
            .map(|c| c.gcs_url)
    }
    /// Enqueue a file for background upload.
    ///
    /// Copies the source file to the queue directory (reflink on APFS/btrfs).
    pub async fn enqueue_file(
        &self,
        source_path: &Path,
        gcs_path: &str,
        content_type: &str,
        artifact_name: &str,
        session_id: &str,
        turn_number: u64,
    ) -> anyhow::Result<()> {
        let in_flight = if is_content_addressed(gcs_path) {
            match self.mark_in_flight(gcs_path) {
                Some(guard) => Some(guard),

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Check worker task logs/panics — wrap the worker loop so panics are caught and completions are always sent (even as errors).
2. Ensure shutdown logic drains in-flight items and completes them (or fails them explicitly) instead of dropping completion_tx.
3. Retry the upload with a fresh queue instance if the worker died during shutdown.
4. Fall back to inline upload when the completion channel errors.

Example fix

// before (worker)
let result = process(item).await;
// if this panics, completion_tx is dropped silently
// after
let result = std::panic::AssertUnwindSafe(process(item).await)
    .catch_unwind()
    .unwrap_or_else(|_| Err(anyhow!("worker panicked")));
let _ = item.completion_tx.send(result.map(|c| c.gcs_url));

fn is_worker_dropped(err: &anyhow::Error) -> bool {
    err.chain().any(|c| c.to_string().contains("worker dropped completion channel"))
}

match queue.enqueue_upload(item).await {
    Ok(url) => url,
    Err(e) if is_worker_dropped(&e) => {
        tracing::error!("upload worker died; retrying once");
        retry_enqueue_or_inline(item).await?
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Enqueue succeeds, but the worker task that should send the completion through `rejected.completion_tx` is dropped/aborted before sending — worker panic, task abort during shutdown, or the worker loop exiting early.

Common situations: Worker task panics on a poisoned/inaccessible queue directory; runtime shutdown aborting background tasks mid-flight; upload cancellation paths that drop the item without sending a completion.

Related errors

• upload queue closed
• No upload credentials. Run `grok login` or set a deployment
• No trace upload bucket configured. Set `GROK_TELEMETRY_GCS_B
• auth entry '{scope_key}' not found while persisting refreshe
• Failed to parse upload response: {}

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/f7a38642f1be1bcf. Report an issue: GitHub.