{"record":{"id":"e9fed2c6530b8323","repo":"zed-industries/zed","slug":"failed-to-generate-evaluation-example-for-commit","errorCode":null,"errorMessage":"failed to generate evaluation example for commit {} at line {} (sample {}): {}","messagePattern":"failed to generate evaluation example for commit (.+?) at line (.+?) \\(sample (.+?)\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/edit_prediction_cli/src/split_commit.rs","lineNumber":548,"sourceCode":"                        Err(e) => {\n                            let err_msg = format!(\n                                \"failed to generate evaluation example for commit {} at line {} (sample {}): {}\",\n                                annotated.commit_sha,\n                                line_num + 1,\n                                sample_idx,\n                                e\n                            );\n                            if e.is::<NoMatchingSplitPointError>() {\n                                eprintln!(\"skipping: {}\", err_msg);\n                                continue;\n                            }\n                            match failed {\n                                FailedHandling::Skip | FailedHandling::SkipNoFiles => {\n                                    eprintln!(\"{}\", err_msg);\n                                    continue;\n                                }\n                                FailedHandling::Keep => {\n                                    anyhow::bail!(err_msg);\n                                }\n                            }\n                        }\n                    };\n\n                    let json = if args.pretty {\n                        serde_json::to_string_pretty(&case)\n                    } else {\n                        serde_json::to_string(&case)\n                    }\n                    .context(\"failed to serialize evaluation case as JSON\")?;\n\n                    // Only add unique samples (different split points may produce same result)\n                    if seen_samples.insert(json.clone()) {\n                        output_lines.push(json);\n                    }\n                }\n            } else {","sourceCodeStart":530,"sourceCodeEnd":566,"githubUrl":"https://github.com/zed-industries/zed/blob/f4178619acd0d47ea1f76a2025c42962c6d6638c/crates/edit_prediction_cli/src/split_commit.rs#L530-L566","documentation":"Bailed in the multi-sample loop of `ep split-commit` (`--num-samples` path) when generating an evaluation example for one commit fails and `--failed` is `keep` — which is the default. With keep, a per-sample generation error aborts the entire run; `no split point found matching <kind>` (NoMatchingSplitPointError) is always skipped regardless of the setting. The inner error explaining why generation failed is appended to the message.","triggerScenarios":"Running `ep split-commit --num-samples N` with `--failed keep` (default) over an annotated-commit JSONL that contains a commit whose reordered diff cannot be turned into an evaluation case (cursor sampling failure, unparseable diff, truncated content).","commonSituations":"Large noisy git histories containing merge/binary/rename commits; curated runs where you intentionally stop on the first bad commit to investigate it.","solutions":["Read the trailing inner error in the message — it names the actual generation failure for that commit and line","Re-run with `--failed skip` to log and skip bad samples instead of aborting the whole run","If you intentionally use keep, fix or remove the offending commit from the input JSONL and re-run"],"exampleFix":"# before (default --failed keep aborts the run)\nep split-commit --num-samples 4 input.jsonl\n\n# after\nep split-commit --num-samples 4 --failed skip input.jsonl","handlingStrategy":"fallback","validationCode":null,"typeGuard":"fn is_expected_split_failure(err: &anyhow::Error) -> bool {\n    err.is::<NoMatchingSplitPointError>() // safe to skip: commit has no matching split point\n}","tryCatchPattern":"Downcast the per-sample error: NoMatchingSplitPointError is always skippable; for other errors honor the configured --failed mode (skip to continue, keep to abort and investigate).","preventionTips":["Use --failed skip during exploratory runs over untrusted commit data","Reserve --failed keep for curated runs where you want the first bad commit to stop the line","Pre-filter input JSONL for commits with parseable, non-binary diffs"],"tags":["dataset-generation","edit-prediction","cli","git"],"backgroundTag":"sample-generation-failed","analyzedSha":"f4178619acd0d47ea1f76a2025c42962c6d6638c","analyzedAt":"2026-08-20T19:29:52.058Z","contentChangedAt":"2026-08-20T19:29:52.058Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}