Test replay failed for '{}' ({}): {}. File deleted - re-run

Error message

Test replay failed for '{}' ({}): {}. File deleted - re-run test to record fresh data.

What it means

A recorded replay file exists for this test+provider, but TestProvider::new_replaying could not load it — typically corrupt JSON, a truncated file from an interrupted run, or a recording format that changed between goose versions. The runner deletes the broken file and tells you to re-run so it records fresh interactions (which requires real credentials).

Source

Thrown at crates/goose-cli/src/scenario_tests/scenario_runner.rs:171

    let file_path = format!(
        "{}/{}/recordings/{}/{}.json",
        manifest_dir,
        SCENARIO_TESTS_DIR,
        factory_name.to_lowercase(),
        test_name
    );

    if let Some(parent) = Path::new(&file_path).parent() {
        std::fs::create_dir_all(parent)?;
    }

    let replay_mode = Path::new(&file_path).exists();
    let (provider_arc, provider_for_saving, original_env) = if replay_mode {
        match TestProvider::new_replaying(&file_path) {
            Ok(test_provider) => (Arc::new(test_provider), None, None),
            Err(e) => {
                let _ = std::fs::remove_file(&file_path);
                return Err(anyhow::anyhow!(
                    "Test replay failed for '{}' ({}): {}. File deleted - re-run test to record fresh data.",
                    test_name, factory_name, e
                ));
            }
        }
    } else {
        if std::env::var("GITHUB_ACTIONS").is_ok() {
            panic!(
                "Test recording is not supported on CI. \
            Did you forget to add the file {} to the repository and were expecting that to replay?",
                file_path
            );
        }

        let original_env = setup_environment(config)?;

        let inner_provider = create(&factory_name, Vec::new()).await?;

View on GitHub (pinned to 3810898a74)

Solutions

Simply re-run the test locally with valid provider credentials — it will record a fresh file; then commit the new recording.
Check `git status` / `git diff` for the deleted file and stage the replacement.
If recordings keep breaking, verify the checkout isn't rewriting line endings (.gitattributes eol settings).

Example fix

# before
$ cargo test scenario_x
Error: Test replay failed for 'scenario_x' (openai): expected value at line 1 column 1. File deleted - re-run test to record fresh data.

# after
$ OPENAI_API_KEY=sk-... cargo test scenario_x   # re-records
$ git add crates/goose-cli/src/scenario_tests/recordings/scenario_x.openai.recorded

Defensive patterns

Strategy: retry

Validate before calling

# validate the recording is loadable JSON before running the suite
for f in $(git ls-files '*.recorded'); do
    jq -e . "$f" >/dev/null || echo "corrupt recording: $f"
done

Try / catch

// the runner already deletes the file; the retry IS the fix
match run_scenario_test(&name).await {
    Err(e) if e.to_string().contains("re-run test to record fresh data") => {
        record_fresh(&name).await?; // needs credentials; then commit the file
    }
    r => r,
}

Prevention

Always commit regenerated recordings after goose version bumps.
Never interrupt a recording run mid-write; discard suspiciously small .recorded files.
Disable CRLF conversion for *.recorded via .gitattributes.

When it happens

Trigger: Upgrading goose when the on-disk recording schema changed; a test process killed mid-write leaving a partial file; a git merge conflict resolved badly inside a .recorded file.

Common situations: Pulling main after a replay-format migration; locally interrupted cargo test runs; CRLF conversion on Windows checkouts corrupting recording files.

Related errors

AI-assisted analysis of aaif-goose/goose@3810898a74 (2026-08-16). Data as JSON: /api/errors/74742dcc4ad7d2f4. Report an issue: GitHub.