{"record":{"id":"45c2b7dc6177e162","repo":"linera-io/linera-protocol","slug":"poisonedworker","errorCode":"PoisonedWorker","errorMessage":"Chain worker was poisoned by a journal resolution failure","messagePattern":"Chain worker was poisoned by a journal resolution failure","errorType":"exception","errorClass":"WorkerError","httpStatus":null,"severity":"critical","filePath":"linera-core/src/chain_worker/state.rs","lineNumber":357,"sourceCode":"        } else {\n            vec![]\n        })\n    }\n\n    /// Returns whether this chain is known to be active (initialized).\n    pub(crate) fn knows_chain_is_active(&self) -> bool {\n        self.knows_chain_is_active\n    }\n\n    /// Rolls back any uncommitted changes to the chain state.\n    pub(crate) fn rollback(&mut self) {\n        self.chain.rollback();\n    }\n\n    /// Returns `WorkerError::PoisonedWorker` if the worker is poisoned due to a database\n    /// `save` failure.\n    pub(crate) fn check_not_poisoned(&self) -> Result<(), WorkerError> {\n        ensure!(!self.poisoned, WorkerError::PoisonedWorker);\n        Ok(())\n    }\n\n    /// Updates the last-access timestamp to the current time.\n    pub(crate) fn touch(&self) {\n        self.last_access.store_now();\n    }\n\n    /// Returns a clone of the last-access `Arc`, for use by the keep-alive task.\n    pub(crate) fn last_access_arc(&self) -> Arc<AtomicTimestamp> {\n        Arc::clone(&self.last_access)\n    }\n\n    /// Drops the service runtime endpoint, signaling the runtime task to stop.\n    /// Returns the runtime task so the caller can await it outside the lock.\n    pub(crate) fn clear_service_runtime(&mut self) -> Option<web_thread_pool::Task<()>> {\n        self.service_runtime_endpoint.take();\n        self.service_runtime_task.take()","sourceCodeStart":339,"sourceCodeEnd":375,"githubUrl":"https://github.com/linera-io/linera-protocol/blob/6c226ddcb332ef55118dc8d0aafbd093d5420899/linera-core/src/chain_worker/state.rs#L339-L375","documentation":"If persisting a chain state change fails during journal resolution (a failed save), the chain worker marks itself poisoned and every subsequent read_lock/write_lock call returns PoisonedWorker. Poisoning is deliberate fail-stop behavior: it prevents the worker from continuing to serve or persist state after the database was left inconsistent by a failed write.","triggerScenarios":"Any subsequent operation on a chain worker after an earlier self.save() failed — the underlying cause is a storage error such as a full disk, corrupted table, or a ScyllaDB/DynamoDB/ServiceStorage outage during block processing.","commonSituations":"Storage backend outage or disk exhaustion during high load; after the first save failure, every request for that chain fails with PoisonedWorker until the process restarts; container memory/disk limits hit in deployments.","solutions":["Fix the underlying storage problem first (free disk space, restore DB availability, check credentials/limits)","Restart the validator node or worker process so chain state is reloaded from the last consistent snapshot","Alert on the first save failure — poisoning always follows one, so monitoring saves gives earlier warning than monitoring PoisonedWorker"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_poisoned_worker(e: &WorkerError) -> bool {\n    matches!(e, WorkerError::PoisonedWorker)\n}","tryCatchPattern":"match worker.handle_block_certificate(cert).await {\n    Err(e) if matches!(e, WorkerError::PoisonedWorker) => {\n        // Fail-stop: do NOT retry in-process. Alert ops and restart the node\n        // after fixing the underlying storage failure.\n        tracing::error!(chain = %chain_id, \"chain worker poisoned after save failure\");\n        return Err(e);\n    }\n    other => other?,\n}","preventionTips":["Monitor storage health (disk space, DB error rates) — poisoning always follows a failed save","Restart the worker process to recover; poisoned state is intentionally not recoverable in-process","Run databases within safe capacity margins and with working failover"],"tags":["storage","database","fail-stop","worker","linera"],"backgroundTag":"poisoned-state-after-write-failure","analyzedSha":"6c226ddcb332ef55118dc8d0aafbd093d5420899","analyzedAt":"2026-08-22T22:49:09.787Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}