theonedev/onedev · error · ExplicitException

Unexpected end of pod watching

Error message

Unexpected end of pod watching

What it means

KubernetesUtils.watchPod streams pod events/logs via kubectl until the watched condition (container start/stop) is met or an abort is signaled. If the underlying watch process ends normally without the condition being observed and without an abort reason, the code treats that as an anomaly and throws this ExplicitException so callers like waitForContainerStart/waitForContainerStop fail fast instead of silently returning.

Source

Thrown at server-core/src/main/java/io/onedev/server/util/KubernetesUtils.java:355

								for (JsonNode containerStatusNode : containerStatusesNode)
									containerStatusNodes.add(containerStatusNode);
							}
							abortRef.set(abortChecker.check(nodeName, containerStatusNodes));
							if (abortRef.get() != null) 
								thread.interrupt();
						}
					}

				}, new LineConsumer() {

					@Override
					public void consume(String line) {
						logKubernetesError(taskLogger, line);
					}

				}).checkReturnCode();

				throw new ExplicitException("Unexpected end of pod watching");
			} catch (Exception e) {
				PodWatchAbort abort = abortRef.get();
				if (abort != null) {
					if (abort.getErrorMessage() != null)
						throw new ExplicitException(abort.getErrorMessage());
					else 
						break;
				} else if (ExceptionUtils.find(e, TimeoutException.class) == null) {
					throw ExceptionUtils.unchecked(e);
				}
			}
		}
	}

	public static void logKubernetesError(TaskLogger taskLogger, String message) {
		if (!message.contains("Failed to watch *unstructured.Unstructured: unknown")) {
			if (message.startsWith("Warning:"))
				taskLogger.warning("Kubernetes: " + message);

View on GitHub (pinned to d44925c47c)

Solutions

1. Check pod events and status (kubectl describe pod / kubectl get pod -o yaml) to see why the container never reached the expected state
2. Retry the build/job — transient node or API-server disruptions often resolve on re-run
3. Inspect the task log lines captured by logKubernetesError for the container's own error output
4. Increase resource requests/limits or node capacity if pods are being evicted; check cluster events for eviction causes

Example fix

// before
waitForContainerStart(podName, containerName, abortRef);
// after
try {
    waitForContainerStart(podName, containerName, abortRef);
} catch (ExplicitException e) {
    logger.warning("pod watch ended before container start: " + e.getMessage());
    // re-create the pod or fail the step with a clear message
}

// before waiting, confirm the pod is still scheduled
kubectl get pod <podName> -o jsonpath='{.status.phase}'  # must not be Failed/Unknown

try {
    KubernetesUtils.waitForContainerStart(podName, containerName, abortRef);
} catch (ExplicitException e) {
    // pod disappeared or watch ended early; inspect and retry once
    throw new RuntimeException("Pod watch ended prematurely: " + e.getMessage(), e);
}

When it happens

Trigger: Calling waitForContainerStart or waitForContainerStop against a pod when the kubectl watch command terminates (non-zero return checked via checkReturnCode, or stream ends) before the container reaches the expected state; the pod may have been deleted, evicted, or the node died mid-watch.

Common situations: Pod evicted or OOM-killed during a CI job; node reboot or cluster upgrade kills the pod; kubectl connection to the API server drops; pod deleted by another controller (e.g. Job backoff limit) while OneDev waits on it.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Allocated agent not connected to current server, please retr
• Attachment not found:
• Malformed build spec
• This build is not authorized to sync to project:
• Loopback address not allowed for target docker image of push

AI-assisted analysis of theonedev/onedev@d44925c47c (2026-09-06). Data as JSON: /api/errors/0ce5fecdbb273944. Report an issue: GitHub.