huggingface/open-r1 · critical · PistonError

All endpoints are unhealthy. Please check your Piston worker

Error message

All endpoints are unhealthy. Please check your Piston workers.

What it means

During failure handling, send_execute probes an endpoint with get_supported_runtimes(); if the probe fails the endpoint is added to _unhealthy_endpoints. Once every configured endpoint is unhealthy, _check_failed_endpoint raises PistonError — the client has exhausted all workers and cannot execute submissions.

Source

Thrown at src/open_r1/utils/competitive_programming/piston_client.py:135

    async def uninstall_package(self, language, version):
        return await self._send_to_all("packages", {"language": language, "version": version}, method="delete")

    async def get_supported_runtimes(self):
        return await self._send_to_all("runtimes", method="get")

    async def _check_failed_endpoint(self, endpoint):
        async with self._endpoint_failures_lock:
            if endpoint in self._unhealthy_endpoints:
                return
            try:
                await asyncio.sleep(5)
                await self.get_supported_runtimes()
            except Exception as e:
                print(f"Error checking endpoint {endpoint}, dropping it ({e})")
                self._unhealthy_endpoints.add(endpoint)
                if len(self._unhealthy_endpoints) >= len(self.base_endpoints):
                    raise PistonError("All endpoints are unhealthy. Please check your Piston workers.")

    async def send_execute(self, data, language="cms_ioi", max_retries=5):
        data = data | {
            "language": language,
            "version": "*",
        }

        base_delay = 1.0

        status = None
        endpoint = None

        for attempt in range(max_retries + 1):
            try:
                endpoint = await self._wait_for_endpoint()
                if attempt > 0:
                    await asyncio.sleep(1)
                async with self.session.post(

View on GitHub (pinned to 1416fa0cf2)

Solutions

Check each endpoint: curl <endpoint>/api/v2/runtimes — restart any worker that doesn't respond.
Verify network connectivity/firewall rules between the client host and the Piston worker ports.
Add more endpoints to PISTON_ENDPOINTS for redundancy and restart evaluation.
Investigate worker logs for crashes (OOM, package corruption) and fix the root cause before resuming.

Example fix

// before: single fragile endpoint
PISTON_ENDPOINTS=http://worker-1:2000
// after: redundant endpoints
PISTON_ENDPOINTS=http://worker-1:2000,http://worker-2:2000,http://worker-3:2000

Defensive patterns

Strategy: retry

Validate before calling

import asyncio, httpx
async def healthy(endpoint):
    try:
        r = await client_http.get(f'{endpoint}/api/v2/runtimes', timeout=5)
        return r.status_code == 200
    except Exception:
        return False
assert await healthy('http://worker-1:2000'), 'endpoint unreachable before run'

Type guard

def any_healthy(endpoints, statuses: dict) -> bool:
    return any(statuses.get(e) is True for e in endpoints)

Try / catch

try:
    score, feedback = await client.send_execute(data)
except PistonError as e:
    if 'All endpoints are unhealthy' in str(e):
        await asyncio.sleep(60)          # wait for workers to recover
        client._unhealthy_endpoints.clear()  # or rebuild the client
        score, feedback = await client.send_execute(data)
    else:
        raise

Prevention

Configure at least 2-3 endpoints for redundancy
Pre-flight health-check all endpoints before large evaluation jobs
Monitor workers (memory, restarts) during parallel execution
Keep PISTON_ENDPOINTS in sync with actual running workers (e.g. via service discovery)

When it happens

Trigger: send_execute repeatedly fails/retries and every endpoint in base_endpoints eventually fails its health check (get_supported_runtimes raising), so len(_unhealthy_endpoints) >= len(base_endpoints).

Common situations: All Piston workers crashed or restarted simultaneously; network/firewall outage between the training node and workers; workers OOM or overloaded during large parallel evaluations; stale PISTON_ENDPOINTS pointing at decommissioned hosts.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/afe43d5266f33b67. Report an issue: GitHub.