agentscope-ai/agentscope · error · RuntimeError

PVC {pvc_name!r} did not finish deleting within {timeout}s

Error message

PVC {pvc_name!r} did not finish deleting within {timeout}s

What it means

Thrown by the K8s workspace when a PersistentVolumeClaim was asked to delete but still exists (not 404) after polling once per second until the timeout. Kubernetes PVC deletion can block on finalizers (e.g. kubernetes.io/pvc-protection) or volumes still attached to a running Pod.

Source

Thrown at src/agentscope/workspace/_k8s/_k8s_workspace.py:342

        pvc_name: str,
        timeout: float = 60.0,
    ) -> None:
        """Poll until the PVC is fully deleted."""
        from kubernetes_asyncio.client.rest import ApiException

        deadline = asyncio.get_event_loop().time() + timeout
        while asyncio.get_event_loop().time() < deadline:
            try:
                await self._v1.read_namespaced_persistent_volume_claim(
                    pvc_name,
                    self._namespace,
                )
            except ApiException as e:
                if e.status == 404:
                    return
                raise
            await asyncio.sleep(1.0)
        raise RuntimeError(
            f"PVC {pvc_name!r} did not finish deleting " f"within {timeout}s",
        )

    async def _ensure_pod(self) -> None:
        """Create or reuse the workspace Pod.

        Only terminal phases (``Failed``, ``Unknown``, ``Succeeded``)
        trigger a delete-and-recreate cycle.  ``Pending`` Pods are left
        for :meth:`_wait_pod_running` which inspects container statuses
        for early failure detection.

        A Pod whose ``metadata.deletion_timestamp`` is set is already
        being deleted (by a previous ``close()`` or an external actor)
        — attaching to it would race the terminator, so we wait for it
        to disappear and then create a fresh one.
        """
        from kubernetes_asyncio.client.rest import ApiException

View on GitHub (pinned to e90f1c7592)

Solutions

1. kubectl get pvc <name> -o yaml — inspect metadata.finalizers and check for pods still mounting it
2. Force-remove the Pod using the PVC (kubectl delete pod --force) or patch away the pvc-protection finalizer if you know it is safe
3. Increase the deletion timeout passed to the workspace
4. Verify the CSI driver for the StorageClass is installed and healthy

Example fix

# before
ws = K8sWorkspace(...)  # RuntimeError: PVC did not finish deleting within 60s

# after
# diagnose the stuck finalizer
# kubectl get pvc ws-agent-123 -o jsonpath='{.metadata.finalizers}'
# kubectl patch pvc ws-agent-123 -p '{"metadata":{"finalizers":null}}'
ws = K8sWorkspace(..., pvc_delete_timeout=300.0)

from kubernetes.client import CoreV1Api
v1 = CoreV1Api()
pvc = v1.read_namespaced_persistent_volume_claim(name, ns)
stuck = [f for f in pvc.metadata.finalizers or []]  # non-empty => deletion will hang

null

try:
    await ws.start()
except RuntimeError as e:
    if "did not finish deleting" in str(e) and "PVC" in str(e):
        # unstick finalizer, then retry
        await asyncio.sleep(5)
        await ws.start()
    else:
        raise

When it happens

Trigger: _ensure_pvc recreating the workspace and waiting for the old PVC to disappear; a finalizer stuck because a Pod using the PVC is still terminating; the volume driver (in-tree or CSI) failing to detach/delete the underlying volume.

Common situations: Node crash leaving the volume attached; CSI driver bugs or missing driver in clusters where the StorageClass provisions slowly; reclaim policies or third-party finalizers (backup tools) holding the PVC.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Credential {credential_id!r} for owner {owner_id!r} disappea
• module 'agentscope.app.storage' has no attribute {name!r}
• This storage backend has no channel support.
• Vector search index {self._index_name!r} on collection {coll
• Ripgrep search timed out after {timeout} seconds. Try search

AI-assisted analysis of agentscope-ai/agentscope@e90f1c7592 (2026-08-28). Data as JSON: /api/errors/a7d0ddfd53b45a84. Report an issue: GitHub.