{"record":{"id":"afe43d5266f33b67","repo":"huggingface/open-r1","slug":"all-endpoints-are-unhealthy-please-check-your-pis","errorCode":null,"errorMessage":"All endpoints are unhealthy. Please check your Piston workers.","messagePattern":"All endpoints are unhealthy\\. Please check your Piston workers\\.","errorType":"exception","errorClass":"PistonError","httpStatus":null,"severity":"critical","filePath":"src/open_r1/utils/competitive_programming/piston_client.py","lineNumber":135,"sourceCode":"\n    async def uninstall_package(self, language, version):\n        return await self._send_to_all(\"packages\", {\"language\": language, \"version\": version}, method=\"delete\")\n\n    async def get_supported_runtimes(self):\n        return await self._send_to_all(\"runtimes\", method=\"get\")\n\n    async def _check_failed_endpoint(self, endpoint):\n        async with self._endpoint_failures_lock:\n            if endpoint in self._unhealthy_endpoints:\n                return\n            try:\n                await asyncio.sleep(5)\n                await self.get_supported_runtimes()\n            except Exception as e:\n                print(f\"Error checking endpoint {endpoint}, dropping it ({e})\")\n                self._unhealthy_endpoints.add(endpoint)\n                if len(self._unhealthy_endpoints) >= len(self.base_endpoints):\n                    raise PistonError(\"All endpoints are unhealthy. Please check your Piston workers.\")\n\n    async def send_execute(self, data, language=\"cms_ioi\", max_retries=5):\n        data = data | {\n            \"language\": language,\n            \"version\": \"*\",\n        }\n\n        base_delay = 1.0\n\n        status = None\n        endpoint = None\n\n        for attempt in range(max_retries + 1):\n            try:\n                endpoint = await self._wait_for_endpoint()\n                if attempt > 0:\n                    await asyncio.sleep(1)\n                async with self.session.post(","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2083b399a2a0bbddd7f6e9563/src/open_r1/utils/competitive_programming/piston_client.py#L117-L153","documentation":"During failure handling, send_execute probes an endpoint with get_supported_runtimes(); if the probe fails the endpoint is added to _unhealthy_endpoints. Once every configured endpoint is unhealthy, _check_failed_endpoint raises PistonError — the client has exhausted all workers and cannot execute submissions.","triggerScenarios":"send_execute repeatedly fails/retries and every endpoint in base_endpoints eventually fails its health check (get_supported_runtimes raising), so len(_unhealthy_endpoints) >= len(base_endpoints).","commonSituations":"All Piston workers crashed or restarted simultaneously; network/firewall outage between the training node and workers; workers OOM or overloaded during large parallel evaluations; stale PISTON_ENDPOINTS pointing at decommissioned hosts.","solutions":["Check each endpoint: curl <endpoint>/api/v2/runtimes — restart any worker that doesn't respond.","Verify network connectivity/firewall rules between the client host and the Piston worker ports.","Add more endpoints to PISTON_ENDPOINTS for redundancy and restart evaluation.","Investigate worker logs for crashes (OOM, package corruption) and fix the root cause before resuming."],"exampleFix":"// before: single fragile endpoint\nPISTON_ENDPOINTS=http://worker-1:2000\n// after: redundant endpoints\nPISTON_ENDPOINTS=http://worker-1:2000,http://worker-2:2000,http://worker-3:2000","handlingStrategy":"retry","validationCode":"import asyncio, httpx\nasync def healthy(endpoint):\n    try:\n        r = await client_http.get(f'{endpoint}/api/v2/runtimes', timeout=5)\n        return r.status_code == 200\n    except Exception:\n        return False\nassert await healthy('http://worker-1:2000'), 'endpoint unreachable before run'","typeGuard":"def any_healthy(endpoints, statuses: dict) -> bool:\n    return any(statuses.get(e) is True for e in endpoints)","tryCatchPattern":"try:\n    score, feedback = await client.send_execute(data)\nexcept PistonError as e:\n    if 'All endpoints are unhealthy' in str(e):\n        await asyncio.sleep(60)          # wait for workers to recover\n        client._unhealthy_endpoints.clear()  # or rebuild the client\n        score, feedback = await client.send_execute(data)\n    else:\n        raise","preventionTips":["Configure at least 2-3 endpoints for redundancy","Pre-flight health-check all endpoints before large evaluation jobs","Monitor workers (memory, restarts) during parallel execution","Keep PISTON_ENDPOINTS in sync with actual running workers (e.g. via service discovery)"],"tags":["piston","network","worker-health","retry-exhausted"],"backgroundTag":"connection-refused","analyzedSha":"1416fa0cf21595d2083b399a2a0bbddd7f6e9563","analyzedAt":"2026-08-30T08:56:53.400Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}