nautechsystems/nautilus_trader · error

Lighter client_order_index probe exhausted after {} attempts

Error message

Lighter client_order_index probe exhausted after {} attempts for cloid {cloid}

What it means

Raised when resolving a unique client_order_index for a cloid fails after exhausting the probe limit. The dispatcher probes candidate indices (skipping ones already bound to other cloids) up to CLOID_INDEX_PROBE_LIMIT + 1 attempts; if every candidate collides, it bails with this message.

Source

Thrown at crates/adapters/lighter/src/websocket/dispatch.rs:1278

                    entry.insert(cloid);

                    if attempt > 0 {
                        log::warn!(
                            "Lighter client_order_index collision at {index}: \
                             cloid {cloid} re-derived to {candidate} after {attempt} probe(s)",
                        );
                    }
                    return Ok(candidate);
                }
                dashmap::mapref::entry::Entry::Occupied(entry) => {
                    if *entry.get() == cloid {
                        return Ok(candidate);
                    }
                    candidate = next_probe_index(candidate);
                }
            }
        }
        anyhow::bail!(
            "Lighter client_order_index probe exhausted after {} attempts for cloid {cloid}",
            CLOID_INDEX_PROBE_LIMIT + 1,
        )
    }

    /// Restore an exact order identity observed during reconciliation.
    ///
    /// `terminal` reflects the current venue report because the cached order can be stale after a
    /// restart.
    ///
    /// # Errors
    ///
    /// Returns an error when the cached order does not carry the same venue order ID, the client
    /// index is outside the venue-safe range, or the binding conflicts with existing local state.
    pub(crate) fn restore_reconciled_order(
        &self,
        order: &OrderAny,
        client_order_index: i64,

View on GitHub (pinned to 18893faf8b)

Solutions

1. Cancel stale open orders and allow reconciliation to prune bindings, freeing probe space.
2. Ensure only one adapter instance owns order submission for the account; multiple instances race for the same index space.
3. Restart the data/execution client to rebuild the binding map from exchange state.
4. Check for a leak where bindings aren't removed after fills/cancels; report/patch the cleanup path.
5. Increase CLOID_INDEX_PROBE_LIMIT if legitimately high order throughput needs more candidates.

Example fix

// before: many stale bindings exhaust probes
anyhow::bail!("probe exhausted after {} attempts for cloid {cloid}", LIMIT + 1);

// after: prune dead bindings before probing
cleanup_resolved_bindings(&self.index_map);
let idx = self.probe_free_index(cloid)?;

// before submitting, ensure a single writer owns the index space
assert!(submission_lock.is_held_by_current_task());

match resolve_client_order_index(cloid).await {
    Err(e) if e.to_string().contains("probe exhausted") => {
        log::error!("index space saturated: {e:#}");
        // trigger reconciliation/restart to rebuild bindings
    }
    r => r?,
}

When it happens

Trigger: Submitting an order with a cloid when the local client_order_index space for the account is saturated with bindings — every probed index is already claimed by an active or recorded order, so no free candidate is found within the limit.

Common situations: Long-running sessions accumulating thousands of live order bindings; a reconciliation bug leaving stale bindings in the index map; reusing the same account across multiple adapter instances that each bind indices independently; an internal map not being cleaned up after cancels.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• AX requires whole contract quantities, was {}
• Order quantity must be at least 1 contract
• Unsupported order type: {:?}, the Architect AX adapter accep
• active Lighter binding conflicts at client_order_index {clie
• missing trigger price for Derive trigger order {}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/fa473739f9c439db. Report an issue: GitHub.