nautechsystems/nautilus_trader · error · DydxError::Nautilus

Broadcast semaphore closed: {e}

Error message

Broadcast semaphore closed: {e}

What it means

Before broadcasting, `broadcast_with_retry` acquires a tokio semaphore permit to serialize transaction broadcasts (ensuring sequence N is broadcast before N+1 is allocated). If `semaphore.acquire()` returns `AcquireError` — which happens only when the semaphore has been closed — the error is wrapped as 'Broadcast semaphore closed'. A closed semaphore means the broadcaster is shutting down or was closed via `close()`, so no further broadcasts can proceed.

Source

Thrown at crates/adapters/dydx/src/execution/broadcaster.rs:180

    ///
    /// # Returns
    ///
    /// The transaction hash on success.
    ///
    /// # Errors
    ///
    /// Returns error if all retries are exhausted or a non-retryable error occurs.
    pub async fn broadcast_with_retry(
        &self,
        tx_manager: &TransactionManager,
        msgs: Vec<Any>,
        operation_name: &str,
    ) -> Result<String, DydxError> {
        // Acquire semaphore to serialize broadcasts.
        // This ensures sequence N is fully broadcast before sequence N+1 is allocated.
        let _permit =
            self.broadcast_semaphore.acquire().await.map_err(|e| {
                DydxError::Nautilus(anyhow::anyhow!("Broadcast semaphore closed: {e}"))
            })?;

        log::debug!("Acquired broadcast permit for {operation_name}");

        // Flag to track if we need to resync sequence before the next attempt.
        // Set by should_retry when a sequence mismatch is detected.
        let needs_resync = Arc::new(AtomicBool::new(false));
        let needs_resync_for_retry = Arc::clone(&needs_resync);

        // Clone values that need to be moved into closures
        let grpc_client = self.grpc_client.clone();
        let rate_limiter = Arc::clone(&self.rate_limiter);
        let op_name = operation_name.to_string();

        let operation = || {
            // Clone captures for the async block
            let needs_resync = Arc::clone(&needs_resync);
            let grpc_client = grpc_client.clone();

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Ensure no broadcasts are submitted after the adapter's stop/disconnect lifecycle has begun
  2. Check application code for calling `close()` on the semaphore/broadcaster while workers are still active
  3. Gracefully drain pending broadcasts before initiating shutdown
  4. Treat this error as terminal: do not retry the broadcast; re-establish the execution client if needed

Example fix

// before
// shutdown path
tokio::spawn(async move { broadcaster.broadcast_with_retry("order", op).await; }); // races close
// after
broadcaster.broadcast_with_retry("order", op).await?; // complete work first
broadcaster.shutdown().await; // then close semaphore
Defensive patterns

Strategy: try-catch

Validate before calling

// Guard against broadcasting during shutdown
if shutting_down.load(std::sync::atomic::Ordering::Acquire) { anyhow::bail!("shutting down, skipping broadcast"); }

Type guard

fn broadcaster_open(b: &Broadcaster) -> bool { !b.is_closed() }

Try / catch

match broadcaster.broadcast_with_retry("order", op).await {
    Err(e) if e.to_string().contains("semaphore closed") => log::warn!("broadcast skipped: shutting down"),
    other => other?,
}

Prevention

When it happens

Trigger: Calling `broadcast_with_retry` after another task called `broadcast_semaphore.close()` — typically during adapter shutdown/stop, or a raced shutdown where a broadcast is attempted concurrently with teardown.

Common situations: Sending a final order/cancel during engine shutdown, a task holding a reference to the broadcaster after `disconnect()`/`stop()`, or reconnect logic racing with a close.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/3fc9dacdedca87a9. Report an issue: GitHub.