JuliusBrussee/caveman · error

cachebench: replay schedule interrupted: %w

Error message

cachebench: replay schedule interrupted: %w

What it means

In sequential replay mode, the sleep until a record's scheduled send time returned an error. The sleep function (injectable, context-aware) fails most commonly because the context was canceled while waiting, and the runner wraps that error rather than sending the request off-schedule.

Source

Thrown at cacheengine/cachebench/replay.go:565

		}
	}
	for provider := range providers {
		if eligible[provider] < target.MinEligibleRequest {
			return fmt.Errorf("cachebench: provider %q engine-eligible population %d cannot meet minimum %d", provider, eligible[provider], target.MinEligibleRequest)
		}
	}
	return nil
}

func (runner ReplayRunner) runSequential(ctx context.Context, prepared []preparedReplay, anchorTrace, anchorReal time.Time, now func() time.Time, sleep func(context.Context, time.Duration) error, emit func(ReplayResult) error) error {
	for _, item := range prepared {
		record := item.record
		traceAt, _ := time.Parse(time.RFC3339Nano, record.At)
		offset, _ := scaledReplayGap(traceAt.Sub(anchorTrace), runner.TimeScale)
		scheduled := anchorReal.Add(offset)
		if delay := scheduled.Sub(now().UTC()); delay > 0 {
			if err := sleep(ctx, delay); err != nil {
				return fmt.Errorf("cachebench: replay schedule interrupted: %w", err)
			}
		}
		started := now().UTC()
		evidence := replayEvidenceBase(record, item.optimized, runner.TimeScale, scheduled, started, runner.Limits.MaxScheduleDrift)
		if runner.Limits.RequireGroundedTiming && evidence.ScheduleDriftMilliseconds > evidence.ScheduleToleranceMilliseconds {
			evidence.FailureCode = "schedule_drift"
			evidence.CompletedAt = now().UTC().Format(time.RFC3339Nano)
			if err := emitValidatedReplayResult(emit, ReplayResult{Evidence: evidence}); err != nil {
				return err
			}
			return &ReplayRunError{
				RequestID: record.RequestID, FailureCode: evidence.FailureCode,
				Err: fmt.Errorf("cachebench: request %q exceeded schedule drift tolerance", record.RequestID),
			}
		}
		result, runErr := runner.executePrepared(ctx, item, evidence, started, now)
		if err := emitValidatedReplayResult(emit, result); err != nil {
			return err

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Verify with errors.Is(err, context.Canceled) / context.DeadlineExceeded whether this is an intentional abort or a timeout.
2. Increase the run deadline or use a TimeScale < 1 to compress gaps so the replay fits the context budget.
3. If it was intentional cancellation, handle it as a clean shutdown and persist results emitted so far.
4. For custom sleep functions, make sure they only return on context done or elapsed time, not spurious errors.

Example fix

// before
runner.TimeScale = 1.0 // real-time gaps exceed a 60s run deadline
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)

// after
runner.TimeScale = 0.01 // compress gaps 100x
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)

if err := ctx.Err(); err != nil {
    return err // do not start a replay with an already-doomed context
}

if err := runner.Run(ctx, records, emit); err != nil {
    if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
        // aborted mid-schedule: results emitted so far are valid; resume or exit
    }
    return err
}

When it happens

Trigger: runSequential waiting for the gap to the next record's scaled timestamp when ctx (or the sleep's context) is canceled or times out; a custom sleep implementation returning its own error; a TimeScale that stretches gaps past the run's deadline.

Common situations: Ctrl+C during a replay with long inter-request gaps; a run-level timeout shorter than the scaled trace duration; TimeScale=1.0 (real-time pacing) on a multi-hour trace under a minutes-long deadline.

Related errors

• cachebench: replay preparation interrupted: %w
• schedule_drift
• cachebench: session %q request %d: %w
• cachebench: observation request %q body digest mismatch
• cachebench: replay request %q has empty provider

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/b91b2d9882d1736b. Report an issue: GitHub.