vxcontrol/pentagi · warning

edit_file was called but did not produce the requested chang

Error message

edit_file was called but did not produce the requested change

What it means

This error comes from the file-edit integration test case in the provider tester. The test drives the model through a scripted tool exchange: the model must call read_file on a known path, then call edit_file with a unified diff that, when applied, replaces an old line with a new line. The error means the model DID call edit_file, but the resulting content failed the verification check — either ApplyUnifiedDiff failed to produce output containing the expected new line, the old line was still present, or the diff targeted the wrong path/produced no change.

Source

Thrown at backend/pkg/providers/tester/file_edit.go:212

		Type:    f.Type(),
		Group:   f.Group(),
		Latency: latency,
	}

	if _, ok := response.(*llms.ContentResponse); !ok {
		result.Error = fmt.Errorf("expected *llms.ContentResponse, got %T", response)
		return result
	}

	switch {
	case f.failure != "":
		result.Error = fmt.Errorf("%s", f.failure)
	case !f.readFileSeen:
		result.Error = fmt.Errorf("model never called the %q tool", tools.FileToolName)
	case !f.editFileSeen:
		result.Error = fmt.Errorf("model called read_file but never followed up with edit_file")
	case !f.editApplied:
		result.Error = fmt.Errorf("edit_file was called but did not produce the requested change")
	default:
		result.Success = true
	}

	return result
}

// firstFileToolCall returns the first tool call in resp targeting PentAGI's
// file tool (tools.FileToolName) along with its decoded arguments.
func firstFileToolCall(resp *llms.ContentResponse) (llms.ToolCall, map[string]any, bool) {
	for _, choice := range resp.Choices {
		for _, call := range choice.ToolCalls {
			if call.FunctionCall == nil || call.FunctionCall.Name != tools.FileToolName {
				continue
			}

			var args map[string]any
			if err := json.Unmarshal([]byte(call.FunctionCall.Arguments), &args); err != nil {

View on GitHub (pinned to ea665308ba)

Solutions

1. Re-run the test — LLM diff generation is nondeterministic, a flaky model may pass on retry.
2. Use a stronger model for the agent under test; diff generation quality correlates strongly with model capability.
3. Check f.failure / the tool-exchange log (appendToolExchange records 'edit_file's diff applied but did not produce ...') to see the exact resulting content the diff produced.
4. Verify the prompt/expected-line constants (FileEditTestOldLine / FileEditTestNewLine) are unambiguous so the diff has exactly one valid form.
5. Improve the edit_file tool prompt/description shown to the model so diff format expectations are explicit.

Example fix

// before (typical failing model output: fuzzy replacement instead of a unified diff)
{"name":"edit_file","arguments":{"path":"/test/file.txt","diff":"replace old line with new line"}}
// after (correct unified diff the test can apply)
{"name":"edit_file","arguments":{"path":"/test/file.txt","diff":"--- a/file.txt\n+++ b/file.txt\n@@ -1,1 +1,1 @@\n-old line\n+new line"}}

// before running the test, confirm the expected transformation is applyable
newContent, _, err := tools.ApplyUnifiedDiff(FileEditTestContent, modelDiff)
if err == nil && strings.Contains(newContent, FileEditTestNewLine) && !strings.Contains(newContent, FileEditTestOldLine) {
    // diff is valid, safe to run the test case
}

func diffProducesExpectedChange(oldContent, diff, wantLine, staleLine string) bool {
    newContent, _, err := tools.ApplyUnifiedDiff(oldContent, diff)
    return err == nil && strings.Contains(newContent, wantLine) && !strings.Contains(newContent, staleLine)
}

result := testCase.Execute(response, latency)
if result.Error != nil {
    if strings.Contains(result.Error.Error(), "did not produce the requested change") {
        log.Printf("model diff invalid, retrying with stronger model: %v", result.Error)
        result = rerunWithModel(ctx, strongerModel)
    }
}

When it happens

Trigger: The model invokes the file tool with name edit_file against FileEditTestPath, but the diff it generates does not transform FileEditTestContent into content containing FileEditTestNewLine and lacking FileEditTestOldLine (f.editApplied stays false, file_edit.go:154-157), so Execute() at file_edit.go:212 reports the failure. Also triggered when the diff applies but is a no-op or only partially matches the context lines.

Common situations: Weaker models emitting malformed or approximate unified diff hunks (wrong context lines, wrong line numbers); models editing a differently-cased or differently-formatted copy of the target line; models producing a diff for a different file path; provider updates (new model versions) that regress tool-calling diff fidelity.

Related errors

• failed to test agent: %w
• failed to test provider: %w
• failed to switch provider: %w
• failed to set provider: %w
• failed to perform agent chain for subtask %d: %w

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/c014f2f1f9f54746. Report an issue: GitHub.