{"record":{"id":"e16ad918a91912cb","repo":"zeroclaw-labs/zeroclaw","slug":"tracellmprovider-turn-turn-index-scripted","errorCode":null,"errorMessage":"TraceLlmProvider({}): turn {turn_index} scripted {leftover} step(s) the agent never requested — the trace over-specifies this turn's LLM round-trips","messagePattern":"TraceLlmProvider\\((.+?)\\): turn (.+?) scripted (.+?) step\\(s\\) the agent never requested — the trace over-specifies this turn's LLM round-trips","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/zeroclaw-eval/src/replay.rs","lineNumber":65,"sourceCode":"    }\n}\n\n/// Runner-side handle for advancing the replay cursor between conversation turns.\n/// Shares the provider's queues (the same `Arc` the agent holds), so the runner can\n/// assert per-turn consumption without owning the boxed provider.\npub struct ReplayHandle {\n    state: Arc<Mutex<ReplayState>>,\n    trace_name: String,\n}\n\nimpl ReplayHandle {\n    /// Assert the just-finished turn consumed all of its scripted steps, then advance\n    /// the cursor to the next turn. Errors if any steps were left unconsumed.\n    pub fn finish_turn(&self, turn_index: usize) -> anyhow::Result<()> {\n        let mut state = self.state.lock().unwrap();\n        let leftover = state.turns.get(state.current).map_or(0, |q| q.len());\n        if leftover > 0 {\n            anyhow::bail!(\n                \"TraceLlmProvider({}): turn {turn_index} scripted {leftover} step(s) the agent never requested — the trace over-specifies this turn's LLM round-trips\",\n                self.trace_name\n            );\n        }\n        state.current += 1;\n        Ok(())\n    }\n}\n\nimpl Attributable for TraceLlmProvider {\n    fn role(&self) -> Role {\n        Role::Provider(ProviderKind::Model(ModelProviderKind::Custom))\n    }\n\n    fn alias(&self) -> &str {\n        \"eval-replay\"\n    }\n}","sourceCodeStart":47,"sourceCodeEnd":83,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-eval/src/replay.rs#L47-L83","documentation":"TraceLlmProvider scripts each turn as a queue of fake LLM round-trips (text and tool-call steps). `finish_turn` asserts the just-finished turn consumed every scripted step; leftover steps mean the agent made fewer `chat()` calls than the trace provides for that turn. It signals that the trace fixture and the agent's actual control flow disagree — the trace over-specifies that turn.","triggerScenarios":"A trace JSON turn with N steps while the agent makes fewer than N LLM calls: e.g. the trace scripts a follow-up tool-result response, but the agent produces its final answer without ever issuing that tool call, so the step is never popped from the queue.","commonSituations":"Hand-editing a recorded trace and adding steps 'for completeness'; changing the case's system prompt so the agent skips a tool call; pairing a trace fixture with the wrong case definition.","solutions":["Remove the unconsumed steps from that turn in the trace fixture (the count is reported as `leftover`)","Re-record or regenerate the trace from an actual run of the current agent","If the step is genuinely needed, adjust the case goal/system prompt so the agent really makes that LLM call"],"exampleFix":"// before — turn 0 scripts 3 steps, the agent only makes 2 chat() calls\n\"turns\": [[{\"text\":\"...\"},{\"tool_call\":\"read_sensor\"},{\"text\":\"final\"}]]\n// after — drop the step the agent never requested\n\"turns\": [[{\"text\":\"...\"},{\"tool_call\":\"read_sensor\"}]]","handlingStrategy":"validation","validationCode":"use std::path::Path;\n\n// Flag fixture turns whose step count looks wrong before running the suite.\nfn audit_trace_steps(trace: &zeroclaw_eval::Trace) -> anyhow::Result<()> {\n    for (i, turn) in trace.turns.iter().enumerate() {\n        if turn.is_empty() {\n            anyhow::bail!(\"trace turn {i} has zero scripted steps — fixture is malformed\");\n        }\n    }\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"match provider.finish_turn(i) {\n    Err(e) if e.to_string().contains(\"never requested\") => {\n        // fixture drift: fail the case as a FIXTURE error, not an agent error\n        report.record_fixture_mismatch(case_name, e);\n    }\n    Err(e) => return Err(e),\n    Ok(()) => {}\n}","preventionTips":["Generate traces from real agent runs instead of hand-authoring step lists","Re-record affected traces whenever prompts, tools, or agent loop logic change","Keep one trace per minimal behavior so drift has a small blast radius"],"tags":["eval","trace-replay","test-fixture","zeroclaw-eval"],"backgroundTag":"test-fixture-mismatch","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}