What causes "cachebench: clock regressed while replaying %q" in JuliusBrussee/caveman?

Use a monotonic source for now (wrap time.Now; Go's time.Time carries monotonic reading when obtained via time.Now and compared with Sub).

JuliusBrussee/caveman · error · ReplayRunError

clock_regression

clock_regression

Error message

cachebench: clock regressed while replaying %q

What it means

After Transport.Send returned, the completion timestamp (from the injected now function) was strictly before the start timestamp. The harness detects the impossible negative latency and fails the request with FailureCode clock_regression instead of recording corrupt timing evidence.

Source

Thrown at cacheengine/cachebench/replay.go:686

}

func (runner ReplayRunner) executePrepared(ctx context.Context, item preparedReplay, evidence ReplayEvidenceRecord, started time.Time, now func() time.Time) (ReplayResult, error) {
	record, optimized := item.record, item.optimized
	response, sendErr := runner.Transport.Send(ctx, ReplayOutbound{
		RequestID: record.RequestID, Provider: record.Provider, Model: record.Model,
		Region: record.Region, Endpoint: record.Endpoint, Body: append([]byte(nil), optimized.Body...),
	})
	completed := now().UTC()
	evidence.HTTPStatus = response.StatusCode
	if len(response.Body) > 0 {
		evidence.ProviderEvidenceSHA256 = bodyDigest(response.Body)
	}
	if completed.Before(started) {
		evidence.CompletedAt = started.Format(time.RFC3339Nano)
		evidence.FailureCode = "clock_regression"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: clock regressed while replaying %q", record.RequestID),
		}
	}
	evidence.CompletedAt = completed.Format(time.RFC3339Nano)
	evidence.LatencyMilliseconds = completed.Sub(started).Milliseconds()
	if sendErr != nil {
		evidence.FailureCode = "transport_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider transport failed for %q: %w", record.RequestID, sendErr),
		}
	}
	if !validProviderRequestID(response.ProviderRequestID) {
		evidence.FailureCode = "provider_response_invalid"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned invalid request identity for %q", record.RequestID),
		}
	}

View on GitHub (pinned to 27d5a3981a)

Solutions

Use a monotonic source for now (wrap time.Now; Go's time.Time carries monotonic reading when obtained via time.Now and compared with Sub).
In tests, make the fake clock strictly non-decreasing across started/completed reads.
If now must be wall-clock, tolerate small backward steps by clamping completed to started instead of failing.
Check host clock synchronization (chrony/ntp) if this reproduces on bare metal.

Example fix

// before
now := func() time.Time { return wallClockNow() } // NTP steps it back mid-request

// after
now := func() time.Time { return time.Now().UTC() } // monotonic-annotated clock

Defensive patterns

Strategy: validation

Validate before calling

// monotonic, strictly non-decreasing clock wrapper
var last time.Time
now := func() time.Time {
    t := time.Now().UTC()
    if t.Before(last) { t = last }
    last = t
    return t
}

Type guard

func isClockRegression(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "clock_regression"
}

Try / catch

if err := runner.Run(ctx, records, emit); err != nil {
    if isClockRegression(err) {
        // your injected now went backwards; switch to monotonic clock and re-run
    }
    return err
}

Prevention

Always inject time.Now().UTC() (monotonic-annotated) as the now function in production.
In tests, make fake clocks tick forward on every read; never return a constant time.
Keep hosts NTP-disciplined (slew, not step) when wall clocks feed timing evidence.

When it happens

Trigger: A custom now function backed by a non-monotonic or mock clock whose value decreases between the two calls; NTP stepping a wall clock backwards mid-request when now is wall-clock based; a fake clock in tests advanced incorrectly.

Common situations: Tests injecting a stub clock that returns a fixed or decreasing time; production code passing time.Now while the host clock is stepped backward by NTP; swapping now between runs without resetting.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/8f2416b7c2849619. Report an issue: GitHub.