TraceLlmProvider({}): turn {turn_index} scripted {leftover}

Error message

TraceLlmProvider({}): turn {turn_index} scripted {leftover} step(s) the agent never requested — the trace over-specifies this turn's LLM round-trips

What it means

TraceLlmProvider scripts each turn as a queue of fake LLM round-trips (text and tool-call steps). `finish_turn` asserts the just-finished turn consumed every scripted step; leftover steps mean the agent made fewer `chat()` calls than the trace provides for that turn. It signals that the trace fixture and the agent's actual control flow disagree — the trace over-specifies that turn.

Source

Thrown at crates/zeroclaw-eval/src/replay.rs:65

    }
}

/// Runner-side handle for advancing the replay cursor between conversation turns.
/// Shares the provider's queues (the same `Arc` the agent holds), so the runner can
/// assert per-turn consumption without owning the boxed provider.
pub struct ReplayHandle {
    state: Arc<Mutex<ReplayState>>,
    trace_name: String,
}

impl ReplayHandle {
    /// Assert the just-finished turn consumed all of its scripted steps, then advance
    /// the cursor to the next turn. Errors if any steps were left unconsumed.
    pub fn finish_turn(&self, turn_index: usize) -> anyhow::Result<()> {
        let mut state = self.state.lock().unwrap();
        let leftover = state.turns.get(state.current).map_or(0, |q| q.len());
        if leftover > 0 {
            anyhow::bail!(
                "TraceLlmProvider({}): turn {turn_index} scripted {leftover} step(s) the agent never requested — the trace over-specifies this turn's LLM round-trips",
                self.trace_name
            );
        }
        state.current += 1;
        Ok(())
    }
}

impl Attributable for TraceLlmProvider {
    fn role(&self) -> Role {
        Role::Provider(ProviderKind::Model(ModelProviderKind::Custom))
    }

    fn alias(&self) -> &str {
        "eval-replay"
    }
}

View on GitHub (pinned to 88bb9c8533)

Solutions

Remove the unconsumed steps from that turn in the trace fixture (the count is reported as `leftover`)
Re-record or regenerate the trace from an actual run of the current agent
If the step is genuinely needed, adjust the case goal/system prompt so the agent really makes that LLM call

Example fix

// before — turn 0 scripts 3 steps, the agent only makes 2 chat() calls
"turns": [[{"text":"..."},{"tool_call":"read_sensor"},{"text":"final"}]]
// after — drop the step the agent never requested
"turns": [[{"text":"..."},{"tool_call":"read_sensor"}]]

Defensive patterns

Strategy: validation

Validate before calling

use std::path::Path;

// Flag fixture turns whose step count looks wrong before running the suite.
fn audit_trace_steps(trace: &zeroclaw_eval::Trace) -> anyhow::Result<()> {
    for (i, turn) in trace.turns.iter().enumerate() {
        if turn.is_empty() {
            anyhow::bail!("trace turn {i} has zero scripted steps — fixture is malformed");
        }
    }
    Ok(())
}

Try / catch

match provider.finish_turn(i) {
    Err(e) if e.to_string().contains("never requested") => {
        // fixture drift: fail the case as a FIXTURE error, not an agent error
        report.record_fixture_mismatch(case_name, e);
    }
    Err(e) => return Err(e),
    Ok(()) => {}
}

Prevention

Generate traces from real agent runs instead of hand-authoring step lists
Re-record affected traces whenever prompts, tools, or agent loop logic change
Keep one trace per minimal behavior so drift has a small blast radius

When it happens

Trigger: A trace JSON turn with N steps while the agent makes fewer than N LLM calls: e.g. the trace scripts a follow-up tool-result response, but the agent produces its final answer without ever issuing that tool call, so the step is never popped from the queue.

Common situations: Hand-editing a recorded trace and adding steps 'for completeness'; changing the case's system prompt so the agent skips a tool call; pairing a trace fixture with the wrong case definition.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/e16ad918a91912cb. Report an issue: GitHub.