xai-org/grok-build · error · anyhow::Error

upload queue closed

Error message

upload queue closed

What it means

Raised while enqueuing an upload when the queue was closed concurrently: the enqueue path detected `closed`, removed the rejected item's files, and returned QueueClosed wrapped with 'upload queue closed'. It means the upload queue is shutting down (or already shut down) and will not accept new items, so the caller's file cannot be scheduled for background upload.

Source

Thrown at crates/codegen/xai-file-utils/src/queue.rs:975

            compress: false,
            parent_span: tracing::Span::current(),
            _in_flight: None,
        };
        self.stats.pending.fetch_add(1, Ordering::Relaxed);
        self.stats.pending_bytes.fetch_add(size, Ordering::Relaxed);
        self.stats.enqueued.fetch_add(1, Ordering::Relaxed);
        match self.tx.try_send(item) {
            Ok(()) => self.stats.notify_transition(),
            Err(e) => {
                let closed = matches!(&e, mpsc::error::TrySendError::Closed(_));
                let rejected = e.into_inner();
                self.stats.pending.fetch_sub(1, Ordering::Relaxed);
                self.stats.pending_bytes.fetch_sub(size, Ordering::Relaxed);
                self.stats.enqueued.fetch_sub(1, Ordering::Relaxed);
                self.stats.notify_transition();
                if closed {
                    remove_item_files(&rejected, Some(&self.stats));
                    return Err(anyhow::Error::new(QueueClosed).context("upload queue closed"));
                }
                if let Some(sidecar) = &rejected.sidecar_path {
                    try_remove_temp(sidecar, Some(&self.stats));
                }
                self.stats.enqueue_fallbacks.fetch_add(1, Ordering::Relaxed);
                self.spawn_inline_upload_owned_snapshot(
                    rejected.source.path().to_path_buf(),
                    gcs_path.to_string(),
                    content_type.to_string(),
                    size,
                    rejected.completion_tx,
                );
            }
        }
        rx.await
            .map_err(|_| {
                anyhow::Error::new(QueueClosed).context("worker dropped completion channel")
            })?

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Check the queue's closed/shutdown state before enqueueing and skip the upload (or write directly) if closed.
2. Ensure all enqueue calls are ordered before close()/shutdown in the session teardown sequence.
3. Perform the upload inline (spawn_inline_upload style) as a fallback when the queue rejects the item.
4. If the error appears without intentional shutdown, audit who calls close() and fix the premature shutdown.

Example fix

// before
let url = queue.enqueue_upload(item).await?;
// after
let url = if queue.is_closed() {
    tracing::warn!("upload queue closed; uploading inline");
    inline_upload(&item).await?
} else {
    queue.enqueue_upload(item).await?
};

fn is_queue_closed(err: &anyhow::Error) -> bool {
    err.chain().any(|c| c.to_string().contains("upload queue closed"))
}

match queue.enqueue_upload(item).await {
    Ok(url) => url,
    Err(e) if is_queue_closed(&e) => {
        tracing::warn!("queue shutting down; uploading inline");
        inline_upload(&item).await?
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling enqueue/upload concurrently with queue shutdown — e.g. a session ending calls close()/drop on the queue while another task is mid-enqueue; the close flag flips after stats were updated, so the item is rejected and its temp files cleaned up.

Common situations: Race between a final artifact upload and session teardown; abrupt client shutdown cancelling the queue; a bug where close() is called too early in the lifecycle.

Related errors

• worker dropped completion channel
• git object database semaphore closed
• No upload credentials. Run `grok login` or set a deployment
• No trace upload bucket configured. Set `GROK_TELEMETRY_GCS_B
• auth entry '{scope_key}' not found while persisting refreshe

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/ae7424377b8758fa. Report an issue: GitHub.