{"record":{"id":"a1dce1ac4e9187cd","repo":"vectordotdev/vector","slug":"poisoned-lock","errorCode":null,"errorMessage":"poisoned lock","messagePattern":"poisoned lock","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/codecs/src/decoding/framing/chunked_gelf.rs","lineNumber":452,"sourceCode":"        );\n\n        let chunk_len = chunk.len();\n\n        // A lone chunk is already complete, but it cannot reuse the ID of a pending message.\n        if total_chunks == 1 {\n            if chunk_len > self.max_length {\n                return Err(ChunkedGelfDecoderError::MaxLengthExceed {\n                    message_id,\n                    sequence_number,\n                    length: chunk_len,\n                    max_length: self.max_length,\n                });\n            }\n\n            // Copy before taking the shared-state lock, then keep the lock from the ID check\n            // through return so another decoder clone cannot insert this ID between them.\n            let chunk = Bytes::copy_from_slice(&chunk);\n            let pending = self.state.lock().expect(\"poisoned lock\");\n            if let Some(message_state) = pending.messages.get(&message_id) {\n                return Err(ChunkedGelfDecoderError::TotalChunksMismatch {\n                    message_id,\n                    sequence_number,\n                    original_total_chunks: message_state.total_chunks,\n                    received_total_chunks: total_chunks,\n                });\n            }\n            return Ok(Some(chunk));\n        }\n\n        let mut pending = self.state.lock().expect(\"poisoned lock\");\n\n        let is_new_message = !pending.messages.contains_key(&message_id);\n        // Settle every admission rule before creating state and a timeout task. The count limit\n        // applies only on insert, since rejecting chunks of pending messages would stall them.\n        if is_new_message {\n            if chunk_len > self.max_length {","sourceCodeStart":434,"sourceCodeEnd":470,"githubUrl":"https://github.com/vectordotdev/vector/blob/bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013/lib/codecs/src/decoding/framing/chunked_gelf.rs#L434-L470","documentation":"The chunked GELF decoder guards its shared reassembly state (`self.state: Mutex<...>`) with `.lock().expect(\"poisoned lock\")`. A std::sync::Mutex is poisoned when another thread panicked while holding it, so after the first panic every subsequent caller panics with this message. The panic happens on the early-return path that checks an existing message's total-chunk count.","triggerScenarios":"`decode_chunk` (called from `decode_message`) locking the state mutex after a panic occurred elsewhere while the lock was held — e.g. a panic inside the timeout task or during chunk reassembly while another decoder clone held the lock.","commonSituations":"Any earlier panic in the chunked GELF decoder (or code holding that lock) in a long-running Vector process; subsequent chunks then all fail with this message, masking the original panic.","solutions":["Find and fix the original panic that poisoned the mutex (look for the first panic in the logs)","Upgrade to a non-poisoning lock (parking_lot::Mutex) or handle poisoning with `.lock().unwrap_or_else(|e| e.into_inner())` if the state is recoverable","Restart the Vector process/workflow as a stopgap"],"exampleFix":"// before\nlet pending = self.state.lock().expect(\"poisoned lock\");\n// after\nlet pending = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());","handlingStrategy":"try-catch","validationCode":"// Rust: detect poisoned state before relying on the shared decoder\nlet healthy = !decoder.state.is_poisoned(); // std Mutex::is_poisoned (stabilized 1.86); else track manually","typeGuard":"fn lock_healthy<T>(m: &std::sync::Mutex<T>) -> bool {\n    !m.is_poisoned()\n}","tryCatchPattern":"// catch_unwind around decode calls, then rebuild the decoder\nlet result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| decoder.decode_message(bytes)));\nmatch result {\n    Ok(inner) => inner,\n    Err(_) => { decoder = ChunkedGelfDecoder::new(config); /* retry once */ }\n}","preventionTips":["Fix any panic occurring while the reassembly lock is held — it poisons the mutex for all later calls","Prefer parking_lot::Mutex (non-poisoning) for shared decoder state","Monitor logs for the first panic; 'poisoned lock' is always secondary","Restart/rebuild decoder instances after an observed panic"],"tags":["rust","mutex","poisoned-lock","concurrency","gelf"],"backgroundTag":"lock-poisoned","analyzedSha":"bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013","analyzedAt":"2026-09-16T02:53:35.741Z","contentChangedAt":"2026-09-16T02:53:35.741Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}