{"record":{"id":"0ce5fecdbb273944","repo":"theonedev/onedev","slug":"unexpected-end-of-pod-watching","errorCode":null,"errorMessage":"Unexpected end of pod watching","messagePattern":"Unexpected end of pod watching","errorType":"exception","errorClass":"ExplicitException","httpStatus":null,"severity":"error","filePath":"server-core/src/main/java/io/onedev/server/util/KubernetesUtils.java","lineNumber":355,"sourceCode":"\t\t\t\t\t\t\t\tfor (JsonNode containerStatusNode : containerStatusesNode)\n\t\t\t\t\t\t\t\t\tcontainerStatusNodes.add(containerStatusNode);\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\tabortRef.set(abortChecker.check(nodeName, containerStatusNodes));\n\t\t\t\t\t\t\tif (abortRef.get() != null) \n\t\t\t\t\t\t\t\tthread.interrupt();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\n\t\t\t\t}, new LineConsumer() {\n\n\t\t\t\t\t@Override\n\t\t\t\t\tpublic void consume(String line) {\n\t\t\t\t\t\tlogKubernetesError(taskLogger, line);\n\t\t\t\t\t}\n\n\t\t\t\t}).checkReturnCode();\n\n\t\t\t\tthrow new ExplicitException(\"Unexpected end of pod watching\");\n\t\t\t} catch (Exception e) {\n\t\t\t\tPodWatchAbort abort = abortRef.get();\n\t\t\t\tif (abort != null) {\n\t\t\t\t\tif (abort.getErrorMessage() != null)\n\t\t\t\t\t\tthrow new ExplicitException(abort.getErrorMessage());\n\t\t\t\t\telse \n\t\t\t\t\t\tbreak;\n\t\t\t\t} else if (ExceptionUtils.find(e, TimeoutException.class) == null) {\n\t\t\t\t\tthrow ExceptionUtils.unchecked(e);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tpublic static void logKubernetesError(TaskLogger taskLogger, String message) {\n\t\tif (!message.contains(\"Failed to watch *unstructured.Unstructured: unknown\")) {\n\t\t\tif (message.startsWith(\"Warning:\"))\n\t\t\t\ttaskLogger.warning(\"Kubernetes: \" + message);","sourceCodeStart":337,"sourceCodeEnd":373,"githubUrl":"https://github.com/theonedev/onedev/blob/d44925c47c37992c828ea673a5f9620539bc3ff2/server-core/src/main/java/io/onedev/server/util/KubernetesUtils.java#L337-L373","documentation":"KubernetesUtils.watchPod streams pod events/logs via kubectl until the watched condition (container start/stop) is met or an abort is signaled. If the underlying watch process ends normally without the condition being observed and without an abort reason, the code treats that as an anomaly and throws this ExplicitException so callers like waitForContainerStart/waitForContainerStop fail fast instead of silently returning.","triggerScenarios":"Calling waitForContainerStart or waitForContainerStop against a pod when the kubectl watch command terminates (non-zero return checked via checkReturnCode, or stream ends) before the container reaches the expected state; the pod may have been deleted, evicted, or the node died mid-watch.","commonSituations":"Pod evicted or OOM-killed during a CI job; node reboot or cluster upgrade kills the pod; kubectl connection to the API server drops; pod deleted by another controller (e.g. Job backoff limit) while OneDev waits on it.","solutions":["Check pod events and status (kubectl describe pod / kubectl get pod -o yaml) to see why the container never reached the expected state","Retry the build/job — transient node or API-server disruptions often resolve on re-run","Inspect the task log lines captured by logKubernetesError for the container's own error output","Increase resource requests/limits or node capacity if pods are being evicted; check cluster events for eviction causes"],"exampleFix":"// before\nwaitForContainerStart(podName, containerName, abortRef);\n// after\ntry {\n    waitForContainerStart(podName, containerName, abortRef);\n} catch (ExplicitException e) {\n    logger.warning(\"pod watch ended before container start: \" + e.getMessage());\n    // re-create the pod or fail the step with a clear message\n}","handlingStrategy":"retry","validationCode":"// before waiting, confirm the pod is still scheduled\nkubectl get pod <podName> -o jsonpath='{.status.phase}'  # must not be Failed/Unknown","typeGuard":null,"tryCatchPattern":"try {\n    KubernetesUtils.waitForContainerStart(podName, containerName, abortRef);\n} catch (ExplicitException e) {\n    // pod disappeared or watch ended early; inspect and retry once\n    throw new RuntimeException(\"Pod watch ended prematurely: \" + e.getMessage(), e);\n}","preventionTips":["Monitor cluster events for evictions and node failures during builds","Set pod disruption budgets / avoid node drains during CI windows","Check task logs (logKubernetesError output) for container errors before retrying","Ensure the kubelog/kubectl connectivity to the API server is stable"],"tags":["kubernetes","pod-lifecycle","watch","ci"],"backgroundTag":"http-error-response","analyzedSha":"d44925c47c37992c828ea673a5f9620539bc3ff2","analyzedAt":"2026-09-06T07:18:27.995Z","contentChangedAt":"2026-09-06T07:18:27.995Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}