What causes "should be able to send message to inference_server." in tracel-ai/burn?

Locate and fix the initial autobatcher-thread failure in the logs; this panic is secondary fallout.

should be able to send message to inference_server.

Error message

should be able to send message to inference_server.

What it means

AsyncPolicy::action sends an ActionMessage (observation, determinism flag and reply Sender) to the autobatcher thread and panics if the send fails. Failure means the inference thread's receiver is gone: the thread panicked on an earlier message or exited its recv loop, so this AsyncPolicy handle is dead and the caller cannot obtain an action.

Solutions

Locate and fix the initial autobatcher-thread failure in the logs; this panic is secondary fallout.
Rebuild the AsyncPolicy with AsyncPolicy::new once the server thread is dead.
Prevent inner-policy panics (validate batched observation shapes, handle device errors) to keep the shared thread alive.
Replace reply-send expects in the server with logged errors so one dead client cannot take down the server for all clients.

Example fix

// before
self.inference_state_sender
    .send(InferenceMessage::ActionMessage(item))
    .expect("should be able to send message to inference_server.");
// after
if self.inference_state_sender.send(InferenceMessage::ActionMessage(item)).is_err() {
    log::error!("Inference server thread is down; cannot request action");
    return fallback_action(states); // or surface an error to the caller
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-validate the observation before sending it to the shared batcher
debug_assert_eq!(states.vec.len(), expected_obs_size, "observation size mismatch would break batched inference");

Try / catch

let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| policy.action(states, deterministic)));
match result {
    Ok((action, ctx)) => (action, ctx),
    Err(_) => { rebuild_async_policy(); fallback_action(states) }
}

Prevention

Fix the first autobatcher-thread panic; every send failure means the thread is already dead.
Validate per-agent observation shapes before batching to prevent inner_policy.action panics.
Replace reply-send expects in flush_actions with logged errors so a single dead client cannot crash the server.
Rebuild the AsyncPolicy after any detected thread death instead of reusing stale clones.

When it happens

Trigger: Calling action() after the autobatcher thread terminated: a previous reply-send panic in flush_actions/flush_logits (e.g. an agent stopped waiting for its action), a panic inside inner_policy.action during batched inference, or all clones of the AsyncPolicy having been dropped elsewhere letting the thread exit with RecvError.

Common situations: Multi-agent RL environments sharing one AsyncPolicy where a single agent's misbehaviour (dropped receiver, timeout) crashes the inference server and every remaining agent panics on its next action() call; CUDA OOM in the inner policy ending the thread mid-rollout.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/10d8db3e9e0409be. Report an issue: GitHub.

Appendix: source

Thrown at crates/burn-rl/src/policy/async_policy.rs:290

        action_receiver
            .recv()
            .expect("AsyncPolicy should receive queued probabilities.")
    }

    fn action(
        &mut self,
        states: Self::Observation,
        deterministic: bool,
    ) -> (Self::Action, Vec<Self::ActionContext>) {
        let (action_sender, action_receiver) = std::sync::mpsc::channel();
        let item = ActionItem {
            sender: action_sender,
            inference_state: states,
            deterministic,
        };
        self.inference_state_sender
            .send(InferenceMessage::ActionMessage(item))
            .expect("should be able to send message to inference_server.");
        let action = action_receiver
            .recv()
            .expect("AsyncPolicy should receive queued actions.");
        (action.action, action.context)
    }

    fn update(&mut self, update: Self::PolicyState) {
        self.inference_state_sender
            .send(InferenceMessage::PolicyUpdate(update))
            .expect("AsyncPolicy should be able to send policy state.")
    }

    fn state(&self) -> Self::PolicyState {
        let (sender, receiver) = mpsc::channel();
        self.inference_state_sender
            .send(InferenceMessage::PolicyRequest(sender))
            .expect("should be able to send message to inference_server.");
        receiver

View on GitHub (pinned to d16f7ba2ed)