nautechsystems/nautilus_trader · error

rate limiter decision lock poisoned

Error message

rate limiter decision lock poisoned

What it means

The generic rate limiter (governor-style GCRA) serializes decisions behind decision_lock. add_quota_for_key inserts a new quota for a key and panics on a poisoned decision_lock — i.e. some thread panicked while holding it, most commonly inside quota.test_and_update or plan/commit logic.

Source

Thrown at crates/network/src/ratelimiter/mod.rs:247

        self.clock.advance(by);
    }
}

impl<K, C> RateLimiter<K, C>
where
    K: Hash + Eq + Clone,
    C: Clock,
{
    /// Adds or updates a quota for a specific key.
    ///
    /// # Panics
    ///
    /// Panics if the rate limiter decision mutex is poisoned.
    pub fn add_quota_for_key(&self, key: K, value: Quota) {
        let _guard = self
            .decision_lock
            .lock()
            .expect("rate limiter decision lock poisoned");
        self.gcra.insert(key, Gcra::new(value));
    }

    /// Checks if the given key is allowed under the rate limit.
    ///
    /// # Errors
    ///
    /// Returns `Err(NotUntil)` if the key is rate-limited, indicating when it will be allowed.
    ///
    /// # Panics
    ///
    /// Panics if the rate limiter decision mutex is poisoned.
    pub fn check_key(&self, key: &K) -> Result<(), NotUntil<C::Instant>> {
        let _guard = self
            .decision_lock
            .lock()
            .expect("rate limiter decision lock poisoned");

View on GitHub (pinned to d1527c24af)

Solutions

1. Find the first panic under decision_lock (often a Quota with zero capacity/period passed to Gcra::new) and fix/validate inputs
2. Validate Quota parameters before inserting (period > 0, burst > 0)
3. Treat lock poisoning as recoverable: decision state is best-effort rate limiting, so into_inner() or error-mapping is preferable to a panic

Example fix

// before
let _guard = self.decision_lock.lock().expect("rate limiter decision lock poisoned");
// after
let _guard = self.decision_lock.lock().unwrap_or_else(|e| e.into_inner());

// reject degenerate quotas before registering
fn assert_quota_ok(q: &Quota) {
    assert!(q.replenish_interval_ns() > 0, "quota period must be > 0");
    assert!(q.burst_size().get() > 0, "quota burst must be > 0");
}

When it happens

Trigger: Calling add_quota_for_key (per-connection/per-domain quota registration) after another thread panicked in check_key/await_keys_ready while holding decision_lock — for example a clock or GCRA arithmetic panic.

Common situations: Mostly seen in tests (test_custom_key_quota, test_multiple_keys) when a shared limiter is reused after a failing test poisoned it; in production, a malformed Quota (zero/negative period) causing arithmetic panic under the lock.

Related errors

• instrument update lock poisoned
• Invalid `ConnectionMode` value: {value}
• Failed to create stream
• in-flight mutex poisoned
• wallet balance mutex poisoned

AI-assisted analysis of nautechsystems/nautilus_trader@d1527c24af (2026-08-27). Data as JSON: /api/errors/2abebbdda79e70f3. Report an issue: GitHub.