{"record":{"id":"fdec9eedb6a507d3","repo":"stablyai/orca","slug":"benchmark-workload-exceeded-the-maxworkloadoverr","errorCode":null,"errorMessage":"Benchmark workload exceeded the ${maxWorkloadOverrunMs}ms sampling overrun limit","messagePattern":"Benchmark workload exceeded the (.+?)ms sampling overrun limit","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"config/scripts/idle-cpu-process-sampling.mjs","lineNumber":186,"sourceCode":"  const samples = []\n  let previousSnapshot = null\n  const needsFinalWorkloadSample = () =>\n    workloadSettledAt !== null && (previousSnapshot?.at ?? -Infinity) < workloadSettledAt\n  while (\n    now() <= requestedDeadline ||\n    samples.length === 0 ||\n    !workloadSettled ||\n    needsFinalWorkloadSample()\n  ) {\n    const sampledAt = now()\n    if (workloadError) {\n      throw workloadError\n    }\n    if (\n      (!workloadSettled && sampledAt >= hardDeadline) ||\n      (workloadSettledAt !== null && workloadSettledAt > hardDeadline)\n    ) {\n      throw new Error(\n        `Benchmark workload exceeded the ${maxWorkloadOverrunMs}ms sampling overrun limit`\n      )\n    }\n    const processRows = descendantsOf(readRows(), rootPid)\n    const rawProcesses = processRows.map((row) => ({ ...row, kind: classify(row, rootPid) }))\n    if (previousSnapshot) {\n      const elapsedSeconds = Math.max(0.001, (sampledAt - previousSnapshot.at) / 1000)\n      const previousByPid = new Map(previousSnapshot.processes.map((proc) => [proc.pid, proc]))\n      const processes = rawProcesses.map((row) => {\n        const previous = previousByPid.get(row.pid)\n        const canComputeDelta =\n          typeof row.cpuTimeSeconds === 'number' && typeof previous?.cpuTimeSeconds === 'number'\n        const cpu = canComputeDelta\n          ? Math.max(0, ((row.cpuTimeSeconds - previous.cpuTimeSeconds) / elapsedSeconds) * 100)\n          : row.percentCpu\n        return { ...row, cpu }\n      })\n      samples.push({","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/stablyai/orca/blob/1136503c6a231a16dce8f921f6fadb63d181e8db/config/scripts/idle-cpu-process-sampling.mjs#L168-L204","documentation":"sampleProcessTreeUntilWorkloadsComplete samples a process tree while a workload promise runs, continuing past the requested sampling duration until the workload settles. To bound sampling on a hung workload it computes hardDeadline = requestedDeadline + maxWorkloadOverrunMs (default 120000ms). It throws when the workload is still unsettled at the hard deadline, or when the workload settled but only after the hard deadline. This prevents sampling forever on a deadlocked/hung workload.","triggerScenarios":"workloadPromise neither resolves nor rejects within requestedDurationMs + maxWorkloadOverrunMs; the workload settles so late that workloadSettledAt exceeds hardDeadline.","commonSituations":"The benchmarked workload hangs on the Wayland GPU stall; a deadlock or infinite loop in the sampled process tree; a CI runner frozen/throttled so the workload can't progress.","solutions":["Pass a workloadPromise that has its own internal timeout/rejection so it settles before the hard deadline.","Raise maxWorkloadOverrunMs if the workload legitimately needs more time (only if the hang is transient).","Inspect the workloadError path (thrown earlier) to rule out an actual workload exception being masked.","Investigate the root hang — a real overrun usually indicates a GPU wedge or deadlock, not a too-small budget."],"exampleFix":"// before\nawait sampleProcessTreeUntilWorkloadsComplete({ rootPid, requestedDurationMs, intervalMs, workloadPromise })\n\n// after — give the workload its own deadline so it rejects instead of hanging the sampler\nconst workloadPromise = Promise.race([\n  runWorkload(),\n  delay(requestedDurationMs + 60_000).then(() => { throw new Error('workload self-timeout') })\n])\nawait sampleProcessTreeUntilWorkloadsComplete({ rootPid, requestedDurationMs, intervalMs, workloadPromise, maxWorkloadOverrunMs: 180_000 })","handlingStrategy":"retry","validationCode":"const workloadPromise = Promise.race([\n  runWorkload(),\n  new Promise((_, reject) => setTimeout(() => reject(new Error('workload self-timeout')), requestedDurationMs + 60_000))\n])","typeGuard":null,"tryCatchPattern":"try {\n  result = await sampleProcessTreeUntilWorkloadsComplete({ rootPid, requestedDurationMs, intervalMs, workloadPromise, maxWorkloadOverrunMs })\n} catch (err) {\n  if (err.message.includes('sampling overrun limit')) {\n    log.error(`Workload did not settle within overrun budget; investigate the hang, not the budget`)\n  }\n  throw err\n}","preventionTips":["Always give the workload its own internal timeout so it rejects instead of hanging the sampler.","Treat an overrun as a symptom of a hung/deadlocked workload first; only widen the budget after ruling that out."],"tags":["benchmark","timeout","process-sampling"],"backgroundTag":null,"analyzedSha":"1136503c6a231a16dce8f921f6fadb63d181e8db","analyzedAt":"2026-08-12T23:15:58.167Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}