failed to generate evaluation example for commit {} at line

Error message

failed to generate evaluation example for commit {} at line {} (sample {}): {}

What it means

Bailed in the multi-sample loop of `ep split-commit` (`--num-samples` path) when generating an evaluation example for one commit fails and `--failed` is `keep` — which is the default. With keep, a per-sample generation error aborts the entire run; `no split point found matching <kind>` (NoMatchingSplitPointError) is always skipped regardless of the setting. The inner error explaining why generation failed is appended to the message.

Source

Thrown at crates/edit_prediction_cli/src/split_commit.rs:548

                        Err(e) => {
                            let err_msg = format!(
                                "failed to generate evaluation example for commit {} at line {} (sample {}): {}",
                                annotated.commit_sha,
                                line_num + 1,
                                sample_idx,
                                e
                            );
                            if e.is::<NoMatchingSplitPointError>() {
                                eprintln!("skipping: {}", err_msg);
                                continue;
                            }
                            match failed {
                                FailedHandling::Skip | FailedHandling::SkipNoFiles => {
                                    eprintln!("{}", err_msg);
                                    continue;
                                }
                                FailedHandling::Keep => {
                                    anyhow::bail!(err_msg);
                                }
                            }
                        }
                    };

                    let json = if args.pretty {
                        serde_json::to_string_pretty(&case)
                    } else {
                        serde_json::to_string(&case)
                    }
                    .context("failed to serialize evaluation case as JSON")?;

                    // Only add unique samples (different split points may produce same result)
                    if seen_samples.insert(json.clone()) {
                        output_lines.push(json);
                    }
                }
            } else {

View on GitHub (pinned to f4178619ac)

Solutions

Read the trailing inner error in the message — it names the actual generation failure for that commit and line
Re-run with `--failed skip` to log and skip bad samples instead of aborting the whole run
If you intentionally use keep, fix or remove the offending commit from the input JSONL and re-run

Example fix

# before (default --failed keep aborts the run)
ep split-commit --num-samples 4 input.jsonl

# after
ep split-commit --num-samples 4 --failed skip input.jsonl

Defensive patterns

Strategy: fallback

Type guard

fn is_expected_split_failure(err: &anyhow::Error) -> bool {
    err.is::<NoMatchingSplitPointError>() // safe to skip: commit has no matching split point
}

Try / catch

Downcast the per-sample error: NoMatchingSplitPointError is always skippable; for other errors honor the configured --failed mode (skip to continue, keep to abort and investigate).

Prevention

Use --failed skip during exploratory runs over untrusted commit data
Reserve --failed keep for curated runs where you want the first bad commit to stop the line
Pre-filter input JSONL for commits with parseable, non-binary diffs

When it happens

Trigger: Running `ep split-commit --num-samples N` with `--failed keep` (default) over an annotated-commit JSONL that contains a commit whose reordered diff cannot be turned into an evaluation case (cursor sampling failure, unparseable diff, truncated content).

Common situations: Large noisy git histories containing merge/binary/rename commits; curated runs where you intentionally stop on the first bad commit to investigate it.

Related errors

AI-assisted analysis of zed-industries/zed@f4178619ac (2026-08-20). Data as JSON: /api/errors/e9fed2c6530b8323. Report an issue: GitHub.