JuliusBrussee/caveman · error · ReplayRunError

cachebench: clock regressed while replaying %q

Error message

cachebench: clock regressed while replaying %q

What it means

The completion timestamp captured after the provider call is before the start timestamp — the system clock jumped backwards mid-request. The run fails closed with failure code clock_regression and completion is clamped to the start time.

Source

Thrown at cacheengine/cachebench/replay.go:686

}

func (runner ReplayRunner) executePrepared(ctx context.Context, item preparedReplay, evidence ReplayEvidenceRecord, started time.Time, now func() time.Time) (ReplayResult, error) {
	record, optimized := item.record, item.optimized
	response, sendErr := runner.Transport.Send(ctx, ReplayOutbound{
		RequestID: record.RequestID, Provider: record.Provider, Model: record.Model,
		Region: record.Region, Endpoint: record.Endpoint, Body: append([]byte(nil), optimized.Body...),
	})
	completed := now().UTC()
	evidence.HTTPStatus = response.StatusCode
	if len(response.Body) > 0 {
		evidence.ProviderEvidenceSHA256 = bodyDigest(response.Body)
	}
	if completed.Before(started) {
		evidence.CompletedAt = started.Format(time.RFC3339Nano)
		evidence.FailureCode = "clock_regression"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: clock regressed while replaying %q", record.RequestID),
		}
	}
	evidence.CompletedAt = completed.Format(time.RFC3339Nano)
	evidence.LatencyMilliseconds = completed.Sub(started).Milliseconds()
	if sendErr != nil {
		evidence.FailureCode = "transport_error"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider transport failed for %q: %w", record.RequestID, sendErr),
		}
	}
	if !validProviderRequestID(response.ProviderRequestID) {
		evidence.FailureCode = "provider_response_invalid"
		return ReplayResult{Evidence: evidence, ProviderResponse: append([]byte(nil), response.Body...)}, &ReplayRunError{
			RequestID: record.RequestID, FailureCode: evidence.FailureCode,
			Err: fmt.Errorf("cachebench: provider returned invalid request identity for %q", record.RequestID),
		}
	}

View on GitHub (pinned to 766dce6b13)

Solutions

1. Fix clock synchronisation (NTP) on the replay host
2. Re-run the replay once the clock is stable

When it happens

Trigger: Thrown at cacheengine/cachebench/replay.go:686 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/e126563e6b47161f. Report an issue: GitHub.