linera-io/linera-protocol · critical · WorkerError

PoisonedWorker

PoisonedWorker

Error message

Chain worker was poisoned by a journal resolution failure

What it means

If persisting a chain state change fails during journal resolution (a failed save), the chain worker marks itself poisoned and every subsequent read_lock/write_lock call returns PoisonedWorker. Poisoning is deliberate fail-stop behavior: it prevents the worker from continuing to serve or persist state after the database was left inconsistent by a failed write.

Source

Thrown at linera-core/src/chain_worker/state.rs:357

        } else {
            vec![]
        })
    }

    /// Returns whether this chain is known to be active (initialized).
    pub(crate) fn knows_chain_is_active(&self) -> bool {
        self.knows_chain_is_active
    }

    /// Rolls back any uncommitted changes to the chain state.
    pub(crate) fn rollback(&mut self) {
        self.chain.rollback();
    }

    /// Returns `WorkerError::PoisonedWorker` if the worker is poisoned due to a database
    /// `save` failure.
    pub(crate) fn check_not_poisoned(&self) -> Result<(), WorkerError> {
        ensure!(!self.poisoned, WorkerError::PoisonedWorker);
        Ok(())
    }

    /// Updates the last-access timestamp to the current time.
    pub(crate) fn touch(&self) {
        self.last_access.store_now();
    }

    /// Returns a clone of the last-access `Arc`, for use by the keep-alive task.
    pub(crate) fn last_access_arc(&self) -> Arc<AtomicTimestamp> {
        Arc::clone(&self.last_access)
    }

    /// Drops the service runtime endpoint, signaling the runtime task to stop.
    /// Returns the runtime task so the caller can await it outside the lock.
    pub(crate) fn clear_service_runtime(&mut self) -> Option<web_thread_pool::Task<()>> {
        self.service_runtime_endpoint.take();
        self.service_runtime_task.take()

View on GitHub (pinned to 6c226ddcb3)

Solutions

1. Fix the underlying storage problem first (free disk space, restore DB availability, check credentials/limits)
2. Restart the validator node or worker process so chain state is reloaded from the last consistent snapshot
3. Alert on the first save failure — poisoning always follows one, so monitoring saves gives earlier warning than monitoring PoisonedWorker

fn is_poisoned_worker(e: &WorkerError) -> bool {
    matches!(e, WorkerError::PoisonedWorker)
}

match worker.handle_block_certificate(cert).await {
    Err(e) if matches!(e, WorkerError::PoisonedWorker) => {
        // Fail-stop: do NOT retry in-process. Alert ops and restart the node
        // after fixing the underlying storage failure.
        tracing::error!(chain = %chain_id, "chain worker poisoned after save failure");
        return Err(e);
    }
    other => other?,
}

When it happens

Trigger: Any subsequent operation on a chain worker after an earlier self.save() failed — the underlying cause is a storage error such as a full disk, corrupted table, or a ScyllaDB/DynamoDB/ServiceStorage outage during block processing.

Common situations: Storage backend outage or disk exhaustion during high load; after the first save failure, every request for that chain fails with PoisonedWorker until the process restarts; container memory/disk limits hit in deployments.

Related errors

• Failed to parse {spawn_mode_name} as a spawn_mode
• Failed to find address for {s}. {parse_error}
• Failed to find port for {s}. {parse_error}
• Failed to find parse port {port_str} for {s}. {parse_error}
• Failed to find parse port {port_str} for {s}

AI-assisted analysis of linera-io/linera-protocol@6c226ddcb3 (2026-08-22). Data as JSON: /api/errors/45c2b7dc6177e162. Report an issue: GitHub.