stablyai/orca · error · Error

Benchmark workload exceeded the ${maxWorkloadOverrunMs}ms sa

Error message

Benchmark workload exceeded the ${maxWorkloadOverrunMs}ms sampling overrun limit

What it means

sampleProcessTreeUntilWorkloadsComplete samples a process tree while a workload promise runs, continuing past the requested sampling duration until the workload settles. To bound sampling on a hung workload it computes hardDeadline = requestedDeadline + maxWorkloadOverrunMs (default 120000ms). It throws when the workload is still unsettled at the hard deadline, or when the workload settled but only after the hard deadline. This prevents sampling forever on a deadlocked/hung workload.

Source

Thrown at config/scripts/idle-cpu-process-sampling.mjs:186

  const samples = []
  let previousSnapshot = null
  const needsFinalWorkloadSample = () =>
    workloadSettledAt !== null && (previousSnapshot?.at ?? -Infinity) < workloadSettledAt
  while (
    now() <= requestedDeadline ||
    samples.length === 0 ||
    !workloadSettled ||
    needsFinalWorkloadSample()
  ) {
    const sampledAt = now()
    if (workloadError) {
      throw workloadError
    }
    if (
      (!workloadSettled && sampledAt >= hardDeadline) ||
      (workloadSettledAt !== null && workloadSettledAt > hardDeadline)
    ) {
      throw new Error(
        `Benchmark workload exceeded the ${maxWorkloadOverrunMs}ms sampling overrun limit`
      )
    }
    const processRows = descendantsOf(readRows(), rootPid)
    const rawProcesses = processRows.map((row) => ({ ...row, kind: classify(row, rootPid) }))
    if (previousSnapshot) {
      const elapsedSeconds = Math.max(0.001, (sampledAt - previousSnapshot.at) / 1000)
      const previousByPid = new Map(previousSnapshot.processes.map((proc) => [proc.pid, proc]))
      const processes = rawProcesses.map((row) => {
        const previous = previousByPid.get(row.pid)
        const canComputeDelta =
          typeof row.cpuTimeSeconds === 'number' && typeof previous?.cpuTimeSeconds === 'number'
        const cpu = canComputeDelta
          ? Math.max(0, ((row.cpuTimeSeconds - previous.cpuTimeSeconds) / elapsedSeconds) * 100)
          : row.percentCpu
        return { ...row, cpu }
      })
      samples.push({

View on GitHub (pinned to 1136503c6a)

Solutions

1. Pass a workloadPromise that has its own internal timeout/rejection so it settles before the hard deadline.
2. Raise maxWorkloadOverrunMs if the workload legitimately needs more time (only if the hang is transient).
3. Inspect the workloadError path (thrown earlier) to rule out an actual workload exception being masked.
4. Investigate the root hang — a real overrun usually indicates a GPU wedge or deadlock, not a too-small budget.

Example fix

// before
await sampleProcessTreeUntilWorkloadsComplete({ rootPid, requestedDurationMs, intervalMs, workloadPromise })

// after — give the workload its own deadline so it rejects instead of hanging the sampler
const workloadPromise = Promise.race([
  runWorkload(),
  delay(requestedDurationMs + 60_000).then(() => { throw new Error('workload self-timeout') })
])
await sampleProcessTreeUntilWorkloadsComplete({ rootPid, requestedDurationMs, intervalMs, workloadPromise, maxWorkloadOverrunMs: 180_000 })

const workloadPromise = Promise.race([
  runWorkload(),
  new Promise((_, reject) => setTimeout(() => reject(new Error('workload self-timeout')), requestedDurationMs + 60_000))
])

try {
  result = await sampleProcessTreeUntilWorkloadsComplete({ rootPid, requestedDurationMs, intervalMs, workloadPromise, maxWorkloadOverrunMs })
} catch (err) {
  if (err.message.includes('sampling overrun limit')) {
    log.error(`Workload did not settle within overrun budget; investigate the hang, not the budget`)
  }
  throw err
}

When it happens

Trigger: workloadPromise neither resolves nor rejects within requestedDurationMs + maxWorkloadOverrunMs; the workload settles so late that workloadSettledAt exceeds hardDeadline.

Common situations: The benchmarked workload hangs on the Wayland GPU stall; a deadlock or infinite loop in the sampled process tree; a CI runner frozen/throttled so the workload can't progress.

Related errors

• Timed out waiting for ${description}
• [plain-node-entry-guard] daemon-entry.js did not exit within
• advertised-url-watcher.ts no longer contains \`${marker}\`
• ${name} must be a positive integer, received ${value}
• ORCA_ADVERTISED_URL_BENCH_ROUNDS must be even

AI-assisted analysis of stablyai/orca@1136503c6a (2026-08-12). Data as JSON: /api/errors/fdec9eedb6a507d3. Report an issue: GitHub.