vectordotdev/vector · error

Data poisoned

Error message

Data poisoned

What it means

The remap transform caches compiled VRL programs behind a `std::sync::Mutex`; `.lock().expect("Data poisoned")` panics when that mutex is poisoned, i.e. some thread panicked while holding it during an earlier compile. This site (src/transforms/remap.rs:197) is the cache lookup: once poisoned, every subsequent `compile_vrl_program` call — new transforms at topology build or config reload — panics with the same message.

Source

Thrown at src/transforms/remap.rs:197

            drop_on_abort: self.drop_on_abort,
            reroute_dropped: self.reroute_dropped,
            runtime: self.runtime,
            cache: Mutex::new(Default::default()),
        }
    }
}

impl RemapConfig {
    fn compile_vrl_program(
        &self,
        enrichment_tables: TableRegistry,
        metrics_storage: MetricsStorage,
        merged_schema_definition: schema::Definition,
    ) -> Result<(Program, String, MeaningList)> {
        if let Some((_, res)) = self
            .cache
            .lock()
            .expect("Data poisoned")
            .iter()
            .find(|v| v.0.0 == enrichment_tables && v.0.1 == merged_schema_definition)
        {
            return res.clone().map_err(Into::into);
        }

        let source = match (&self.source, &self.file, &self.files) {
            (Some(source), None, None) => source.to_owned(),
            (None, Some(path), None) => Self::read_file(path)?,
            (None, None, Some(paths)) => {
                let mut combined_source = String::new();
                for path in paths {
                    let content = Self::read_file(path)?;
                    combined_source.push_str(&content);
                    combined_source.push('\n');
                }
                combined_source
            }

View on GitHub (pinned to 99894c8d88)

Solutions

1. Find the first panic preceding the "Data poisoned" message and fix its root cause (often a VRL compiler bug — upgrade Vector)
2. Restart Vector after fixing the root cause to clear the poisoned lock
3. If embedding/patching: use `lock().unwrap_or_else(|e| e.into_inner())` or parking_lot::Mutex (no poisoning) for a cache that should survive unrelated panics

Example fix

// before
self.cache.lock().expect("Data poisoned").iter().find(...)
// after
self.cache
    .lock()
    .unwrap_or_else(|poisoned| poisoned.into_inner())
    .iter()
    .find(...)

# shell: a prior panic precedes every poisoning; surface it first
journalctl -u vector | grep -B5 -m1 'panicked' | head -40

// Rust (embedding): treat a poisoned compile cache as a cache miss, not a crash
let cache = self
    .cache
    .lock()
    .unwrap_or_else(|poisoned| poisoned.into_inner()); // recover the still-valid map

When it happens

Trigger: Any panic inside the critical section (most plausibly a panic during VRL compilation or a downstream expect while the cache lock is held) poisons the mutex; the next remap compilation on the same RemapConfig then hits "Data poisoned" at the lookup.

Common situations: Always the tail of another failure: search earlier in the logs for the original panic. Recurs on every config reload after the first compile-time panic, making Vector appear unable to reload configuration at all.

Related errors

• time ewma gauge mutex poisoned
• mutex poisoned
• event will be set
• argument must be a string
• argument must be a string

AI-assisted analysis of vectordotdev/vector@99894c8d88 (2026-08-20). Data as JSON: /api/errors/267c38889dc234cf. Report an issue: GitHub.