AsyncPolicy should receive queued probabilities.

Error message

AsyncPolicy should receive queued probabilities.

What it means

After forwarding the observation, AsyncPolicy::forward blocks on action_receiver.recv() and panics if it returns Err — which only happens when every Sender for that reply channel was dropped without sending. The autobatcher's flush_logits drops the sender after send, so an Err means flush_logits never ran for this item: the server thread died (panicked or exited) before processing the queued ForwardMessage, leaving the caller stranded.

Source

Thrown at crates/burn-rl/src/policy/async_policy.rs:274

    type ActionContext = P::ActionContext;
    type PolicyState = P::PolicyState;

    type Observation = P::Observation;
    type ActionDistribution = P::ActionDistribution;
    type Action = P::Action;

    fn forward(&mut self, states: Self::Observation) -> Self::ActionDistribution {
        let (action_sender, action_receiver) = std::sync::mpsc::channel();
        let item = ForwardItem {
            sender: action_sender,
            inference_state: states,
        };
        self.inference_state_sender
            .send(InferenceMessage::ForwardMessage(item))
            .expect("Should be able to send message to inference_server");
        action_receiver
            .recv()
            .expect("AsyncPolicy should receive queued probabilities.")
    }

    fn action(
        &mut self,
        states: Self::Observation,
        deterministic: bool,
    ) -> (Self::Action, Vec<Self::ActionContext>) {
        let (action_sender, action_receiver) = std::sync::mpsc::channel();
        let item = ActionItem {
            sender: action_sender,
            inference_state: states,
            deterministic,
        };
        self.inference_state_sender
            .send(InferenceMessage::ActionMessage(item))
            .expect("should be able to send message to inference_server.");
        let action = action_receiver
            .recv()

View on GitHub (pinned to d16f7ba2ed)

Solutions

Check autobatcher-thread logs for the root panic (often inside flush_logits or the inner policy) and fix it.
Verify agent accounting: a wrong increment_agents/decrement_agents balance can stall batches forever; keep the counts consistent with the actual number of concurrent callers.
Add resilience: have the server flush pending items or reply with errors during shutdown instead of dropping senders silently.
Consider recv_timeout on the caller side with a clear error/timeout message rather than an opaque panic on a dead channel.

Example fix

// before
action_receiver
    .recv()
    .expect("AsyncPolicy should receive queued probabilities.")
// after
action_receiver
    .recv_timeout(std::time::Duration::from_secs(30))
    .unwrap_or_else(|err| panic!("No probabilities returned from inference server (server may have died): {}", err))

Defensive patterns

Strategy: retry

Validate before calling

// Check server responsiveness before blocking indefinitely on a queued forward
let probe = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    let mut p = policy.clone();
    let _ = p.forward(sample_observation.clone());
}));
assert!(probe.is_ok(), "inference server not responding");

Try / catch

match action_receiver.recv_timeout(Duration::from_secs(30)) {
    Ok(dist) => dist,
    Err(RecvTimeoutError::Timeout) => return Err(anyhow::anyhow!("inference timed out; batch possibly stalled")),
    Err(RecvTimeoutError::Disconnected) => return Err(anyhow::anyhow!("inference server died before returning probabilities")),
}

Prevention

Use recv_timeout instead of recv to distinguish slow inference from a dead server.
Keep increment_agents/decrement_agents counts accurate so batches always reach the flush threshold.
Fix root panics in flush_logits/inner_policy.forward that strand queued requests.
Prefer graceful server shutdown that flushes or rejects pending items instead of dropping senders.

When it happens

Trigger: The autobatcher thread panicked on an earlier message or inside inner_policy.forward while flushing a batch, so this ForwardItem's reply sender was dropped by unwinding without a value being sent; or the thread exited its loop entirely so the queued ForwardMessage is never processed and the item (with its sender) is leaked/dropped at process exit.

Common situations: Deadlock-then-panic patterns: an agent's request waits behind a batch that can never fill (num_agents misconfigured via increment/decrement), and an unrelated panic kills the server, converting every waiter's recv into this panic; GPU faults during batched forward aborting all pending requests.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/0dcf2be345aacea2. Report an issue: GitHub.