JuliusBrussee/caveman · error · ReplayRunError

schedule_drift

schedule_drift

Error message

cachebench: request %q exceeded schedule drift tolerance

What it means

With Limits.RequireGroundedTiming enabled, a sequential replay request started later than its scheduled time by more than the drift tolerance (derived from Limits.MaxScheduleDrift). The harness fails the request with FailureCode schedule_drift and emits its evidence record, because timing-sensitive measurements are invalid when real send time diverges from the schedule.

Source

Thrown at cacheengine/cachebench/replay.go:578

		traceAt, _ := time.Parse(time.RFC3339Nano, record.At)
		offset, _ := scaledReplayGap(traceAt.Sub(anchorTrace), runner.TimeScale)
		scheduled := anchorReal.Add(offset)
		if delay := scheduled.Sub(now().UTC()); delay > 0 {
			if err := sleep(ctx, delay); err != nil {
				return fmt.Errorf("cachebench: replay schedule interrupted: %w", err)
			}
		}
		started := now().UTC()
		evidence := replayEvidenceBase(record, item.optimized, runner.TimeScale, scheduled, started, runner.Limits.MaxScheduleDrift)
		if runner.Limits.RequireGroundedTiming && evidence.ScheduleDriftMilliseconds > evidence.ScheduleToleranceMilliseconds {
			evidence.FailureCode = "schedule_drift"
			evidence.CompletedAt = now().UTC().Format(time.RFC3339Nano)
			if err := emitValidatedReplayResult(emit, ReplayResult{Evidence: evidence}); err != nil {
				return err
			}
			return &ReplayRunError{
				RequestID: record.RequestID, FailureCode: evidence.FailureCode,
				Err: fmt.Errorf("cachebench: request %q exceeded schedule drift tolerance", record.RequestID),
			}
		}
		result, runErr := runner.executePrepared(ctx, item, evidence, started, now)
		if err := emitValidatedReplayResult(emit, result); err != nil {
			return err
		}
		if runErr != nil {
			return runErr
		}
	}
	return nil
}

func (runner ReplayRunner) runConcurrent(ctx context.Context, prepared []preparedReplay, anchorTrace, anchorReal time.Time, now func() time.Time, sleep func(context.Context, time.Duration) error, emit func(ReplayResult) error) error {
	scheduleCtx, stopScheduling := context.WithCancel(ctx)
	defer stopScheduling()
	semaphore := make(chan struct{}, runner.Limits.MaxConcurrency)
	var workers sync.WaitGroup

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Raise Limits.MaxScheduleDrift to a tolerance the host can realistically honor.
2. Run timing-sensitive replays on an idle machine (no CI neighbors, power saving disabled).
3. If exact timing is not the measurement goal, set Limits.RequireGroundedTiming = false.
4. Use a slower TimeScale (larger gaps) so scheduling jitter is proportionally smaller than the tolerance.

Example fix

// before
limits := cachebench.ReplayLimits{RequireGroundedTiming: true, MaxScheduleDrift: time.Millisecond}

// after
limits := cachebench.ReplayLimits{RequireGroundedTiming: true, MaxScheduleDrift: 50 * time.Millisecond}

// calibrate before the run: assert tolerance comfortably exceeds observed scheduler jitter
if limits.RequireGroundedTiming && limits.MaxScheduleDrift < 10*time.Millisecond {
    return errors.New("drift tolerance too strict for this host; raise MaxScheduleDrift")
}

func isScheduleDrift(err error) bool {
    var rre *cachebench.ReplayRunError
    return errors.As(err, &rre) && rre.FailureCode == "schedule_drift"
}

if err := runner.Run(ctx, records, emit); err != nil {
    if isScheduleDrift(err) {
        // evidence was emitted for the failed request; widen MaxScheduleDrift and re-run
    }
    return err
}

When it happens

Trigger: sleep returning late (scheduler contention, GC pause, overloaded host) so started - scheduled exceeds ScheduleToleranceMilliseconds; a too-tight MaxScheduleDrift for the achievable pacing; TimeScale so small that sub-millisecond scheduling is expected.

Common situations: Running timing-grounded replays on shared CI runners or busy laptops; MaxScheduleDrift left at a strict default while the machine cannot keep up; long traces where occasional pauses accumulate.

Related errors

• cachebench: replay schedule interrupted: %w
• clock_regression
• cachebench: session %q request %d: %w
• cachebench: observation request %q body digest mismatch
• cachebench: replay request %q has empty provider

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/afadd99c5327f47f. Report an issue: GitHub.