{"record":{"id":"f7a38642f1be1bcf","repo":"xai-org/grok-build","slug":"worker-dropped-completion-channel","errorCode":null,"errorMessage":"worker dropped completion channel","messagePattern":"worker dropped completion channel","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-file-utils/src/queue.rs","lineNumber":992,"sourceCode":"                    remove_item_files(&rejected, Some(&self.stats));\n                    return Err(anyhow::Error::new(QueueClosed).context(\"upload queue closed\"));\n                }\n                if let Some(sidecar) = &rejected.sidecar_path {\n                    try_remove_temp(sidecar, Some(&self.stats));\n                }\n                self.stats.enqueue_fallbacks.fetch_add(1, Ordering::Relaxed);\n                self.spawn_inline_upload_owned_snapshot(\n                    rejected.source.path().to_path_buf(),\n                    gcs_path.to_string(),\n                    content_type.to_string(),\n                    size,\n                    rejected.completion_tx,\n                );\n            }\n        }\n        rx.await\n            .map_err(|_| {\n                anyhow::Error::new(QueueClosed).context(\"worker dropped completion channel\")\n            })?\n            .map(|c| c.gcs_url)\n    }\n    /// Enqueue a file for background upload.\n    ///\n    /// Copies the source file to the queue directory (reflink on APFS/btrfs).\n    pub async fn enqueue_file(\n        &self,\n        source_path: &Path,\n        gcs_path: &str,\n        content_type: &str,\n        artifact_name: &str,\n        session_id: &str,\n        turn_number: u64,\n    ) -> anyhow::Result<()> {\n        let in_flight = if is_content_addressed(gcs_path) {\n            match self.mark_in_flight(gcs_path) {\n                Some(guard) => Some(guard),","sourceCodeStart":974,"sourceCodeEnd":1010,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-file-utils/src/queue.rs#L974-L1010","documentation":"Raised when awaiting the oneshot completion channel (`rx.await`) for an enqueued upload and the sender was dropped before replying — meaning the queue worker panicked, was cancelled, or shut down without completing the item. The QueueClosed error is wrapped with 'worker dropped completion channel' to distinguish worker death from an explicit queue-close rejection.","triggerScenarios":"Enqueue succeeds, but the worker task that should send the completion through `rejected.completion_tx` is dropped/aborted before sending — worker panic, task abort during shutdown, or the worker loop exiting early.","commonSituations":"Worker task panics on a poisoned/inaccessible queue directory; runtime shutdown aborting background tasks mid-flight; upload cancellation paths that drop the item without sending a completion.","solutions":["Check worker task logs/panics — wrap the worker loop so panics are caught and completions are always sent (even as errors).","Ensure shutdown logic drains in-flight items and completes them (or fails them explicitly) instead of dropping completion_tx.","Retry the upload with a fresh queue instance if the worker died during shutdown.","Fall back to inline upload when the completion channel errors."],"exampleFix":"// before (worker)\nlet result = process(item).await;\n// if this panics, completion_tx is dropped silently\n// after\nlet result = std::panic::AssertUnwindSafe(process(item).await)\n    .catch_unwind()\n    .unwrap_or_else(|_| Err(anyhow!(\"worker panicked\")));\nlet _ = item.completion_tx.send(result.map(|c| c.gcs_url));","handlingStrategy":"retry","validationCode":null,"typeGuard":"fn is_worker_dropped(err: &anyhow::Error) -> bool {\n    err.chain().any(|c| c.to_string().contains(\"worker dropped completion channel\"))\n}","tryCatchPattern":"match queue.enqueue_upload(item).await {\n    Ok(url) => url,\n    Err(e) if is_worker_dropped(&e) => {\n        tracing::error!(\"upload worker died; retrying once\");\n        retry_enqueue_or_inline(item).await?\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Wrap worker task bodies with panic capture and always send a completion (success or error)","Ensure shutdown drains in-flight items rather than aborting worker tasks","Monitor worker task health (JoinHandle result) and restart on unexpected exit","Avoid dropping items without completing their oneshot senders in cancel paths"],"tags":["upload","queue","worker","channel","concurrency"],"backgroundTag":"worker-dropped-channel","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}