{"record":{"id":"54743dd3d436d7d8","repo":"xai-org/grok-build","slug":"resources-persistence-writer-dropped-acknowledgeme","errorCode":null,"errorMessage":"resources persistence writer dropped acknowledgement","messagePattern":"resources persistence writer dropped acknowledgement","errorType":"exception","errorClass":"SchedulerError::Persistence","httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-grok-tools/src/implementations/grok_build/scheduler/actor.rs","lineNumber":353,"sourceCode":"                .enqueue_save_and_flush(res.serialize());\n            drop(res);\n            tracing::info!(task_id = %task_id, \"Scheduled task expired; removing without firing\");\n\n            let persistence = match acknowledgement {\n                Ok(acknowledgement) => {\n                    let deadline = tokio::time::Instant::now() + DURABILITY_BARRIER_TIMEOUT;\n                    tokio::select! {\n                        _ = self.cancel_token.cancelled() => {\n                            ExpiryPersistenceOutcome::Unknown(SchedulerError::Cancelled)\n                        }\n                        result = tokio::time::timeout_at(deadline, acknowledgement) => {\n                            match result {\n                                Ok(Ok(Ok(()))) => ExpiryPersistenceOutcome::Committed,\n                                Ok(Ok(Err(error))) => {\n                                    ExpiryPersistenceOutcome::NotCommitted(error)\n                                }\n                                Ok(Err(_)) => ExpiryPersistenceOutcome::Unknown(\n                                    SchedulerError::Persistence(std::io::Error::new(\n                                        std::io::ErrorKind::BrokenPipe,\n                                        \"resources persistence writer dropped acknowledgement\",\n                                    )),\n                                ),\n                                Err(_) => {\n                                    ExpiryPersistenceOutcome::Unknown(SchedulerError::Timeout)\n                                }\n                            }\n                        }\n                    }\n                }\n                Err(error) => ExpiryPersistenceOutcome::NotCommitted(error),\n            };\n            match persistence {\n                ExpiryPersistenceOutcome::Committed => {}\n                ExpiryPersistenceOutcome::NotCommitted(error) => {\n                    tracing::warn!(\n                        %task_id,","sourceCodeStart":335,"sourceCodeEnd":371,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-grok-tools/src/implementations/grok_build/scheduler/actor.rs#L335-L371","documentation":"In the scheduler actor's fire_next_task expiry persistence path, when the oneshot channel carrying the writer's acknowledgement is itself dropped without a reply (JoinError on the writer task), the outcome is marked Unknown with a BrokenPipe io::Error \"resources persistence writer dropped acknowledgement\". This means it cannot be determined whether the expiry was durably persisted — the writer task died or was cancelled mid-operation.","triggerScenarios":"The persistence writer task panics or is aborted while handling a SaveAndFlush command, so the oneshot Sender is dropped without sending; the runtime shuts down the writer task before the acknowledgement round-trips.","commonSituations":"Runtime shutdown during graceful stop while expiry persistence is in flight; a panic inside the writer's disk I/O loop; the background writer task being cancelled by a timeout harness (as in the durable-persistence tests).","solutions":["Treat the outcome as Unknown and reconcile on startup: re-derive task absence/presence from disk rather than assuming the write failed.","Inspect writer task logs for a panic/cancel and fix the underlying I/O error that killed it.","Retry the save via a fresh writer instance once it is restarted.","Ensure the writer task is not aborted while save requests are outstanding (drain before shutdown)."],"exampleFix":"// before\nlet outcome = scheduler.fire_next_task(...).await?; // assume committed\n// after\nif let ExpiryPersistenceOutcome::Unknown(err) = outcome {\n    tracing::error!(\"expiry persistence indeterminate: {err}; will reconcile from disk\");\n    self.reconcile_from_disk().await?;\n}","handlingStrategy":"retry","validationCode":"// Before relying on persisted state, confirm the writer task is alive:\nlet writer_alive = writer_handle.is_finished() == false;","typeGuard":null,"tryCatchPattern":"match outcome {\n    ExpiryPersistenceOutcome::Committed => {/* ok */},\n    ExpiryPersistenceOutcome::Unknown(err) => {\n        tracing::error!(\"indeterminate: {err}\");\n        retry_with_backoff(|| persist_expiry(absence)).await?;\n        reconcile_from_disk().await?;\n    }\n    ExpiryPersistenceOutcome::NotCommitted(e) => return Err(e.into()),\n}","preventionTips":["Drain in-flight saves before stopping the writer task.","Catch panics inside the writer loop so one bad write does not kill the task.","Reconcile durability from disk on startup after any Unknown outcome.","Monitor writer task liveness with a health check."],"tags":["persistence","actor","broken-pipe","durability"],"backgroundTag":"persistence-ack-lost","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}