Test replay failed for '{}' ({}) - missing recorded interact

Error message

Test replay failed for '{}' ({}) - missing recorded interaction: {}. File deleted - re-run test to record fresh data.

What it means

Replay mode was active and the recorded provider reported 'No recorded response found' while answering the scenario — the current run made a request (a prompt or tool-call round trip) that has no matching entry in the recording. The runner deletes the file so the next run records fresh data. It means the code path drifted from what was captured: changed prompts, different tool-call counts, or model nondeterminism producing an extra exchange.

Source

Thrown at crates/goose-cli/src/scenario_tests/scenario_runner.rs:281

    )
    .await;

    let mut error = None;
    for message in &messages {
        if let Err(e) = cli_session
            .process_message(message.clone(), CancellationToken::default(), false)
            .await
        {
            error = Some(e.to_string());
            break;
        }
    }
    let updated_messages = cli_session.message_history();

    if let Some(ref err_msg) = error {
        if err_msg.contains("No recorded response found") {
            let _ = std::fs::remove_file(&file_path);
            return Err(anyhow::anyhow!(
                "Test replay failed for '{}' ({}) - missing recorded interaction: {}. File deleted - re-run test to record fresh data.",
                test_name, factory_name, err_msg
            ));
        }
    }

    let result = ScenarioResult {
        messages: updated_messages,
        error,
    };

    validator(&result)?;

    drop(cli_session);

    if let Some(provider) = provider_for_saving {
        if result.error.is_none() {
            Arc::try_unwrap(provider)

View on GitHub (pinned to 3810898a74)

Solutions

Re-run the test locally with credentials to re-record against the current code path, then commit the updated file.
Keep scenario inputs deterministic (fixed prompts, no timestamps/randomness) so recordings stay stable.
If it recurs every run for one provider, make that scenario's message generator deterministic or add the provider to the skip list.

Example fix

# before
$ cargo test scenario_y
Error: Test replay failed for 'scenario_y' (anthropic) - missing recorded interaction: No recorded response found for prompt 5. File deleted - ...

# after
$ ANTHROPIC_API_KEY=... cargo test scenario_y   # fresh recording
$ git commit -am "re-record scenario_y" && git push

Defensive patterns

Strategy: retry

Validate before calling

# keep scenario inputs deterministic so request counts match recordings
export GOOSE_TEST_SEED=fixed  # any project-specific determinism knobs

Try / catch

match run_scenario_test(&name).await {
    Err(e) if e.to_string().contains("missing recorded interaction") => {
        record_fresh(&name).await?; // runner deleted the file; re-record and commit
    }
    r => r,
}

Prevention

Re-record immediately after changing any prompt or recipe used by a scenario.
Keep scenario message generators deterministic (no random IDs/timestamps).
Review .recorded diffs in PRs to catch accidental interaction drift.

When it happens

Trigger: Editing a recipe/prompt after the recording was made; a model update altering how many back-and-forth exchanges occur; nondeterministic agent behavior adding an unrecorded tool call during replay.

Common situations: Recordings made on an older recipe version; contributors changing test prompts without re-recording; providers whose request streams vary run to run.

Related errors

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/eafef3249edec6f8. Report an issue: GitHub.