tracel-ai/burn · error

Should be able to send message to inference_server

Error message

Should be able to send message to inference_server

What it means

AsyncPolicy::forward sends a ForwardMessage containing the observation and a reply Sender to the autobatcher thread, then panics if the send fails. Send failure means the receiving end was closed, i.e. the inference thread has exited (previous panic or RecvError after all handles were dropped). The caller's Policy::forward therefore aborts with a panic instead of returning an ActionDistribution.

Source

Thrown at crates/burn-rl/src/policy/async_policy.rs:271

where
    P: Policy + Send + 'static,
{
    type ActionContext = P::ActionContext;
    type PolicyState = P::PolicyState;

    type Observation = P::Observation;
    type ActionDistribution = P::ActionDistribution;
    type Action = P::Action;

    fn forward(&mut self, states: Self::Observation) -> Self::ActionDistribution {
        let (action_sender, action_receiver) = std::sync::mpsc::channel();
        let item = ForwardItem {
            sender: action_sender,
            inference_state: states,
        };
        self.inference_state_sender
            .send(InferenceMessage::ForwardMessage(item))
            .expect("Should be able to send message to inference_server");
        action_receiver
            .recv()
            .expect("AsyncPolicy should receive queued probabilities.")
    }

    fn action(
        &mut self,
        states: Self::Observation,
        deterministic: bool,
    ) -> (Self::Action, Vec<Self::ActionContext>) {
        let (action_sender, action_receiver) = std::sync::mpsc::channel();
        let item = ActionItem {
            sender: action_sender,
            inference_state: states,
            deterministic,
        };
        self.inference_state_sender
            .send(InferenceMessage::ActionMessage(item))

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Find and fix the original error that terminated the autobatcher thread (check for earlier panic messages in logs).
2. Recreate the AsyncPolicy via AsyncPolicy::new — the channel cannot be reopened once the thread exits.
3. Make inner_policy.forward panic-free (validate inputs, handle device errors) to protect the shared inference thread.
4. In the server, replace reply-send expects with logged warnings so dropped receivers don't kill the thread for everyone.

Example fix

// before
self.inference_state_sender
    .send(InferenceMessage::ForwardMessage(item))
    .expect("Should be able to send message to inference_server");
// after
match self.inference_state_sender.send(InferenceMessage::ForwardMessage(item)) {
    Ok(()) => {}
    Err(err) => {
        log::error!("Inference server unavailable: {}", err);
        return self.inner_policy_fallback_forward(states); // or propagate a proper error
    }
}

// Ensure the server is alive before issuing inference
let alive = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    let mut p = policy.clone();
    let _ = p.forward(sample_observation.clone());
}));
if alive.is_err() { rebuild_async_policy(); }

let dist = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| policy.forward(states)))
    .map_err(|_| anyhow::anyhow!("AsyncPolicy inference server unavailable"))?;

When it happens

Trigger: Calling forward() when the autobatcher thread is dead: a prior panic in the server loop (failed reply send in flush_logits/flush_actions, or a panic inside inner_policy.forward such as a tensor shape/device error), or using a stale AsyncPolicy clone after the thread already exited.

Common situations: Distributed RL rollouts where one worker's dropped reply channel panicked the server and all other workers then panic on their next forward; GPU OOM or batch-shape mismatch inside the inner policy killing the thread mid-episode; hot-reloading policies by dropping old handles while workers still hold clones.

Related errors

• Autobatcher should be able to send current policy state.
• Can send message to autobatcher.
• AsyncPolicy should be able to receive policy state.
• AsyncPolicy should receive queued probabilities.
• should be able to send message to inference_server.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/e6c11cd05bb3cc44. Report an issue: GitHub.