JuliusBrussee/caveman · error · ReplayRunError

model_visible_mismatch

model_visible_mismatch

Error message

cachebench: request %q failed model-visible equivalence

What it means

After optimization was applied, ModelVisibleEquivalent judged the optimized body not equivalent to the native body from the model's point of view. The cache engine is only allowed to rewrite requests in ways invisible to the model, so the run aborts with FailureCode model_visible_mismatch instead of sending a changed request.

Source

Thrown at cacheengine/cachebench/replay.go:522

		if err := ctx.Err(); err != nil {
			return nil, fmt.Errorf("cachebench: replay preparation interrupted: %w", err)
		}
		native, err := record.NativeRequest()
		if err != nil {
			return nil, err
		}
		optimized, err := runner.Engine.Optimize(ctx, native)
		if err != nil {
			return nil, fmt.Errorf("cachebench: optimize request %q: %w", record.RequestID, err)
		}
		equivalent := bytes.Equal(native.Body, optimized.Body)
		if optimized.Applied {
			equivalent = ModelVisibleEquivalent(native.Body, optimized.Body)
		}
		if !equivalent {
			return nil, &ReplayRunError{
				RequestID: record.RequestID, FailureCode: "model_visible_mismatch",
				Err: fmt.Errorf("cachebench: request %q failed model-visible equivalence", record.RequestID),
			}
		}
		if optimized.Decision != cacheengine.DecisionApply && optimized.Decision != cacheengine.DecisionObserveOnly && optimized.Reason != cacheengine.ReasonBelowMinimum {
			return nil, &ReplayRunError{
				RequestID: record.RequestID, FailureCode: "engine_not_cacheable",
				Err: fmt.Errorf("cachebench: request %q is not cacheable: %s", record.RequestID, optimized.Reason),
			}
		}
		prepared = append(prepared, preparedReplay{record: record, optimized: optimized})
	}
	return prepared, nil
}

func validatePreparedReplayTarget(prepared []preparedReplay, target Target) error {
	if err := validateTarget(target); err != nil {
		return err
	}
	providers := map[string]bool{}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Reproduce: run Engine.Optimize on the named RequestID and diff native.Body vs optimized.Body to see exactly what changed.
2. Fix or disable the offending rewrite rule in the engine configuration so only model-invisible transformations are applied.
3. If the rewrite is legitimately safe but the checker is too strict, fix ModelVisibleEquivalent for that construct and add a regression test.
4. As a workaround for measurement-only runs, configure the engine to observe-only mode so bodies are sent unmodified.

Example fix

// before
engine := cacheengine.New(rules) // rule 'drop-system-echo' alters visible content

// after
engine := cacheengine.New(rules.Without("drop-system-echo")) // or fix rule to stay model-invisible

for _, r := range records {
    native, _ := r.NativeRequest()
    opt, err := engine.Optimize(ctx, native)
    if err != nil { return err }
    eq := bytes.Equal(native.Body, opt.Body)
    if opt.Applied { eq = cachebench.ModelVisibleEquivalent(native.Body, opt.Body) }
    if !eq { return fmt.Errorf("record %s would fail equivalence", r.RequestID) }
}

func isModelVisibleMismatch(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "model_visible_mismatch"
}

if err := runner.Run(ctx, records, emit); err != nil {
    if isModelVisibleMismatch(err) {
        // diff native vs optimized body for the named request; disable the offending rule
    }
    return err
}

When it happens

Trigger: An engine rewrite (token pruning, field reordering, padding changes) that alters model-visible content: dropping a non-degenerate message, changing message roles or ordering, altering tool definitions, or rewriting inside prompt text the equivalence checker considers visible.

Common situations: A new optimization rule shipped without equivalence guarantees; a provider schema change making the old equivalence logic flag legitimate rewrites; hand-crafted trace bodies with unusual structure that the equivalence function misclassifies.

Related errors

• cachebench: optimize request %q: %w
• engine_not_cacheable
• registry output has no valid registry_sha256
• registry output has no capabilities
• registry output has invalid capability

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/9a7c744834ab5f5d. Report an issue: GitHub.