{"record":{"id":"1cdc332a0a5f7358","repo":"huggingface/open-r1","slug":"piston-overloaded-res-json-run-stderr","errorCode":null,"errorMessage":"Piston overloaded: {res_json['run']['stderr']}","messagePattern":"Piston overloaded: (.+?)","errorType":"exception","errorClass":"PistonError","httpStatus":null,"severity":"error","filePath":"src/open_r1/utils/competitive_programming/piston_client.py","lineNumber":165,"sourceCode":"\n        for attempt in range(max_retries + 1):\n            try:\n                endpoint = await self._wait_for_endpoint()\n                if attempt > 0:\n                    await asyncio.sleep(1)\n                async with self.session.post(\n                    f\"{endpoint.rstrip('/')}/execute\", json=data, headers={\"Content-Type\": \"application/json\"}\n                ) as response:\n                    status = response.status\n                    res_json = await response.json(content_type=None)\n\n                    if status != 200:\n                        raise PistonError(f\"Server error. status={status}. {res_json}\")\n                    if res_json is None:\n                        raise PistonError(f\"Empty response. status={status}\")\n                    # piston overloaded\n                    if \"run\" in res_json and \"Resource temporarily unavailable\" in res_json[\"run\"].get(\"stderr\", \"\"):\n                        raise PistonError(f\"Piston overloaded: {res_json['run']['stderr']}\")\n                    return res_json\n\n            except (PistonError, asyncio.TimeoutError, aiohttp.ClientConnectionError, RuntimeError) as e:\n                # Only retry if we haven't reached max retries yet\n                if attempt < max_retries:\n                    # Calculate backoff with jitter\n                    delay = min(base_delay * (2**attempt), 10)  # Exponential backoff, capped at 10 seconds\n                    jitter = delay * 0.2 * (2 * asyncio.get_event_loop().time() % 1 - 0.5)  # Add ±10% jitter\n                    retry_delay = delay + jitter\n                    print(f\"Retrying in {retry_delay:.2f} seconds [{self.endpoint_ids[endpoint]}] {endpoint} - {e}\")\n\n                    # special case: worker died\n                    if isinstance(e, aiohttp.ClientConnectionError) and \"Connect call failed\" in str(e):\n                        await self._check_failed_endpoint(endpoint)\n                    else:\n                        # hopefully we won't get this one again\n                        await self._release_endpoint(endpoint)\n                    endpoint = None","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2083b399a2a0bbddd7f6e9563/src/open_r1/utils/competitive_programming/piston_client.py#L147-L183","documentation":"PistonClient.send_execute raises PistonError('Piston overloaded: ...') when a successful execution response contains 'Resource temporarily unavailable' in run.stderr. This means Piston accepted the request but the host OS could not fork/allocate resources (fork failure, out of file descriptors, cgroup limits) to run the sandbox. The client retries automatically, but persistent occurrences mean the worker host is resource-exhausted.","triggerScenarios":"res_json['run']['stderr'] contains 'Resource temporarily unavailable' (EAGAIN) — concurrent executions exceeding the host's process/memory/fd limits, too many sandboxes per worker, low ulimit -u / pid_max.","commonSituations":"Running many parallel scoring workers against a small Piston fleet, container cgroup limits (pids.max) too low, host under memory pressure, too-low RLIMIT_NPROC for the piston user.","solutions":["Reduce client-side concurrency (number of parallel score_single_test_case tasks)","Increase ulimit -u (max user processes) and file descriptors for the piston service","Raise container cgroup pids.max / memory limits for Piston workers","Scale out: add more Piston workers behind the load balancer","Keep the client's built-in retry with backoff; it often clears transient overload"],"exampleFix":"// before\nawait asyncio.gather(*[score_single_test_case(c) for c in cases])  # 200 concurrent\n// after\nsem = asyncio.Semaphore(16)\nawait asyncio.gather(*[score_with_sem(sem, c) for c in cases])","handlingStrategy":"retry","validationCode":"import resource, os\nsoft, hard = resource.getrlimit(resource.RLIMIT_NPROC)\nprint(f\"RLIMIT_NPROC soft={soft} hard={hard}\")  # ensure ample process budget on worker hosts\ndef worker_has_headroom(active, limit):\n    return active < limit * 0.8","typeGuard":"def is_overload_error(e):\n    return isinstance(e, PistonError) and e.args and \"Piston overloaded\" in str(e.args[0])","tryCatchPattern":"try:\n    res = await client.send_execute(data)\nexcept PistonError as e:\n    if \"Piston overloaded\" in str(e):\n        await asyncio.sleep(random.uniform(2, 10))  # extra backoff beyond client retry\n        res = await client.send_execute(data)\n    else:\n        raise","preventionTips":["Bound client concurrency with an asyncio.Semaphore sized to worker capacity","Raise ulimit -u / nofile and container pids.max for piston services","Monitor worker memory and fork-failure rates","Add workers horizontally instead of pushing more concurrency per host"],"tags":["overload","resource-limits","concurrency","sandbox"],"backgroundTag":"resource-temporarily-unavailable","analyzedSha":"1416fa0cf21595d2083b399a2a0bbddd7f6e9563","analyzedAt":"2026-08-30T08:56:53.400Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}