tracel-ai/burn · error

should be able to send message to inference_server.

Error message

should be able to send message to inference_server.

What it means

AsyncPolicy::action sends an ActionMessage (observation, determinism flag and reply Sender) to the autobatcher thread and panics if the send fails. Failure means the inference thread's receiver is gone: the thread panicked on an earlier message or exited its recv loop, so this AsyncPolicy handle is dead and the caller cannot obtain an action.

Source

Thrown at crates/burn-rl/src/policy/async_policy.rs:290

        action_receiver
            .recv()
            .expect("AsyncPolicy should receive queued probabilities.")
    }

    fn action(
        &mut self,
        states: Self::Observation,
        deterministic: bool,
    ) -> (Self::Action, Vec<Self::ActionContext>) {
        let (action_sender, action_receiver) = std::sync::mpsc::channel();
        let item = ActionItem {
            sender: action_sender,
            inference_state: states,
            deterministic,
        };
        self.inference_state_sender
            .send(InferenceMessage::ActionMessage(item))
            .expect("should be able to send message to inference_server.");
        let action = action_receiver
            .recv()
            .expect("AsyncPolicy should receive queued actions.");
        (action.action, action.context)
    }

    fn update(&mut self, update: Self::PolicyState) {
        self.inference_state_sender
            .send(InferenceMessage::PolicyUpdate(update))
            .expect("AsyncPolicy should be able to send policy state.")
    }

    fn state(&self) -> Self::PolicyState {
        let (sender, receiver) = mpsc::channel();
        self.inference_state_sender
            .send(InferenceMessage::PolicyRequest(sender))
            .expect("should be able to send message to inference_server.");
        receiver

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Locate and fix the initial autobatcher-thread failure in the logs; this panic is secondary fallout.
2. Rebuild the AsyncPolicy with AsyncPolicy::new once the server thread is dead.
3. Prevent inner-policy panics (validate batched observation shapes, handle device errors) to keep the shared thread alive.
4. Replace reply-send expects in the server with logged errors so one dead client cannot take down the server for all clients.

Example fix

// before
self.inference_state_sender
    .send(InferenceMessage::ActionMessage(item))
    .expect("should be able to send message to inference_server.");
// after
if self.inference_state_sender.send(InferenceMessage::ActionMessage(item)).is_err() {
    log::error!("Inference server thread is down; cannot request action");
    return fallback_action(states); // or surface an error to the caller
}

// Pre-validate the observation before sending it to the shared batcher
debug_assert_eq!(states.vec.len(), expected_obs_size, "observation size mismatch would break batched inference");

let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| policy.action(states, deterministic)));
match result {
    Ok((action, ctx)) => (action, ctx),
    Err(_) => { rebuild_async_policy(); fallback_action(states) }
}

When it happens

Trigger: Calling action() after the autobatcher thread terminated: a previous reply-send panic in flush_actions/flush_logits (e.g. an agent stopped waiting for its action), a panic inside inner_policy.action during batched inference, or all clones of the AsyncPolicy having been dropped elsewhere letting the thread exit with RecvError.

Common situations: Multi-agent RL environments sharing one AsyncPolicy where a single agent's misbehaviour (dropped receiver, timeout) crashes the inference server and every remaining agent panics on its next action() call; CUDA OOM in the inner policy ending the thread mid-rollout.

Related errors

• AsyncPolicy should receive queued actions.
• Autobatcher should be able to send current policy state.
• Can send message to autobatcher.
• Should be able to send message to inference_server
• AsyncPolicy should receive queued probabilities.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/10d8db3e9e0409be. Report an issue: GitHub.