{"record":{"id":"9a7c744834ab5f5d","repo":"JuliusBrussee/caveman","slug":"model-visible-mismatch","errorCode":"model_visible_mismatch","errorMessage":"cachebench: request %q failed model-visible equivalence","messagePattern":"cachebench: request %q failed model-visible equivalence","errorType":"exception","errorClass":"ReplayRunError","httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/replay.go","lineNumber":522,"sourceCode":"\t\tif err := ctx.Err(); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"cachebench: replay preparation interrupted: %w\", err)\n\t\t}\n\t\tnative, err := record.NativeRequest()\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\toptimized, err := runner.Engine.Optimize(ctx, native)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"cachebench: optimize request %q: %w\", record.RequestID, err)\n\t\t}\n\t\tequivalent := bytes.Equal(native.Body, optimized.Body)\n\t\tif optimized.Applied {\n\t\t\tequivalent = ModelVisibleEquivalent(native.Body, optimized.Body)\n\t\t}\n\t\tif !equivalent {\n\t\t\treturn nil, &ReplayRunError{\n\t\t\t\tRequestID: record.RequestID, FailureCode: \"model_visible_mismatch\",\n\t\t\t\tErr: fmt.Errorf(\"cachebench: request %q failed model-visible equivalence\", record.RequestID),\n\t\t\t}\n\t\t}\n\t\tif optimized.Decision != cacheengine.DecisionApply && optimized.Decision != cacheengine.DecisionObserveOnly && optimized.Reason != cacheengine.ReasonBelowMinimum {\n\t\t\treturn nil, &ReplayRunError{\n\t\t\t\tRequestID: record.RequestID, FailureCode: \"engine_not_cacheable\",\n\t\t\t\tErr: fmt.Errorf(\"cachebench: request %q is not cacheable: %s\", record.RequestID, optimized.Reason),\n\t\t\t}\n\t\t}\n\t\tprepared = append(prepared, preparedReplay{record: record, optimized: optimized})\n\t}\n\treturn prepared, nil\n}\n\nfunc validatePreparedReplayTarget(prepared []preparedReplay, target Target) error {\n\tif err := validateTarget(target); err != nil {\n\t\treturn err\n\t}\n\tproviders := map[string]bool{}","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/replay.go#L504-L540","documentation":"After optimization was applied, ModelVisibleEquivalent judged the optimized body not equivalent to the native body from the model's point of view. The cache engine is only allowed to rewrite requests in ways invisible to the model, so the run aborts with FailureCode model_visible_mismatch instead of sending a changed request.","triggerScenarios":"An engine rewrite (token pruning, field reordering, padding changes) that alters model-visible content: dropping a non-degenerate message, changing message roles or ordering, altering tool definitions, or rewriting inside prompt text the equivalence checker considers visible.","commonSituations":"A new optimization rule shipped without equivalence guarantees; a provider schema change making the old equivalence logic flag legitimate rewrites; hand-crafted trace bodies with unusual structure that the equivalence function misclassifies.","solutions":["Reproduce: run Engine.Optimize on the named RequestID and diff native.Body vs optimized.Body to see exactly what changed.","Fix or disable the offending rewrite rule in the engine configuration so only model-invisible transformations are applied.","If the rewrite is legitimately safe but the checker is too strict, fix ModelVisibleEquivalent for that construct and add a regression test.","As a workaround for measurement-only runs, configure the engine to observe-only mode so bodies are sent unmodified."],"exampleFix":"// before\nengine := cacheengine.New(rules) // rule 'drop-system-echo' alters visible content\n\n// after\nengine := cacheengine.New(rules.Without(\"drop-system-echo\")) // or fix rule to stay model-invisible","handlingStrategy":"validation","validationCode":"for _, r := range records {\n    native, _ := r.NativeRequest()\n    opt, err := engine.Optimize(ctx, native)\n    if err != nil { return err }\n    eq := bytes.Equal(native.Body, opt.Body)\n    if opt.Applied { eq = cachebench.ModelVisibleEquivalent(native.Body, opt.Body) }\n    if !eq { return fmt.Errorf(\"record %s would fail equivalence\", r.RequestID) }\n}","typeGuard":"func isModelVisibleMismatch(err error) bool {\n    var rre *cachebench.ReplayRunError\n    return errors.As(err, &rre) && rre.FailureCode == \"model_visible_mismatch\"\n}","tryCatchPattern":"if err := runner.Run(ctx, records, emit); err != nil {\n    if isModelVisibleMismatch(err) {\n        // diff native vs optimized body for the named request; disable the offending rule\n    }\n    return err\n}","preventionTips":["Unit-test each engine rewrite rule with an equivalence assertion over representative bodies.","Pin engine and trace-capture to compatible versions in CI.","Add regression fixtures for any past model_visible_mismatch."],"tags":["cachebench","equivalence","engine","correctness"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}