huggingface/open-r1 · error · PistonError

Piston overloaded: {res_json['run']['stderr']}

Error message

Piston overloaded: {res_json['run']['stderr']}

What it means

PistonClient.send_execute raises PistonError('Piston overloaded: ...') when a successful execution response contains 'Resource temporarily unavailable' in run.stderr. This means Piston accepted the request but the host OS could not fork/allocate resources (fork failure, out of file descriptors, cgroup limits) to run the sandbox. The client retries automatically, but persistent occurrences mean the worker host is resource-exhausted.

Source

Thrown at src/open_r1/utils/competitive_programming/piston_client.py:165

        for attempt in range(max_retries + 1):
            try:
                endpoint = await self._wait_for_endpoint()
                if attempt > 0:
                    await asyncio.sleep(1)
                async with self.session.post(
                    f"{endpoint.rstrip('/')}/execute", json=data, headers={"Content-Type": "application/json"}
                ) as response:
                    status = response.status
                    res_json = await response.json(content_type=None)

                    if status != 200:
                        raise PistonError(f"Server error. status={status}. {res_json}")
                    if res_json is None:
                        raise PistonError(f"Empty response. status={status}")
                    # piston overloaded
                    if "run" in res_json and "Resource temporarily unavailable" in res_json["run"].get("stderr", ""):
                        raise PistonError(f"Piston overloaded: {res_json['run']['stderr']}")
                    return res_json

            except (PistonError, asyncio.TimeoutError, aiohttp.ClientConnectionError, RuntimeError) as e:
                # Only retry if we haven't reached max retries yet
                if attempt < max_retries:
                    # Calculate backoff with jitter
                    delay = min(base_delay * (2**attempt), 10)  # Exponential backoff, capped at 10 seconds
                    jitter = delay * 0.2 * (2 * asyncio.get_event_loop().time() % 1 - 0.5)  # Add ±10% jitter
                    retry_delay = delay + jitter
                    print(f"Retrying in {retry_delay:.2f} seconds [{self.endpoint_ids[endpoint]}] {endpoint} - {e}")

                    # special case: worker died
                    if isinstance(e, aiohttp.ClientConnectionError) and "Connect call failed" in str(e):
                        await self._check_failed_endpoint(endpoint)
                    else:
                        # hopefully we won't get this one again
                        await self._release_endpoint(endpoint)
                    endpoint = None

View on GitHub (pinned to 1416fa0cf2)

Solutions

1. Reduce client-side concurrency (number of parallel score_single_test_case tasks)
2. Increase ulimit -u (max user processes) and file descriptors for the piston service
3. Raise container cgroup pids.max / memory limits for Piston workers
4. Scale out: add more Piston workers behind the load balancer
5. Keep the client's built-in retry with backoff; it often clears transient overload

Example fix

// before
await asyncio.gather(*[score_single_test_case(c) for c in cases])  # 200 concurrent
// after
sem = asyncio.Semaphore(16)
await asyncio.gather(*[score_with_sem(sem, c) for c in cases])

import resource, os
soft, hard = resource.getrlimit(resource.RLIMIT_NPROC)
print(f"RLIMIT_NPROC soft={soft} hard={hard}")  # ensure ample process budget on worker hosts
def worker_has_headroom(active, limit):
    return active < limit * 0.8

def is_overload_error(e):
    return isinstance(e, PistonError) and e.args and "Piston overloaded" in str(e.args[0])

try:
    res = await client.send_execute(data)
except PistonError as e:
    if "Piston overloaded" in str(e):
        await asyncio.sleep(random.uniform(2, 10))  # extra backoff beyond client retry
        res = await client.send_execute(data)
    else:
        raise

When it happens

Trigger: res_json['run']['stderr'] contains 'Resource temporarily unavailable' (EAGAIN) — concurrent executions exceeding the host's process/memory/fd limits, too many sandboxes per worker, low ulimit -u / pid_max.

Common situations: Running many parallel scoring workers against a small Piston fleet, container cgroup limits (pids.max) too low, host under memory pressure, too-low RLIMIT_NPROC for the piston user.

Related errors

• E2B is not available and required for this provider. Please
• MorphCloud is not available and required for this provider.

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/1cdc332a0a5f7358. Report an issue: GitHub.