{"record":{"id":"a7d0ddfd53b45a84","repo":"agentscope-ai/agentscope","slug":"pvc-pvc-name-r-did-not-finish-deleting-within-t","errorCode":null,"errorMessage":"PVC {pvc_name!r} did not finish deleting within {timeout}s","messagePattern":"PVC (.+?) did not finish deleting within (.+?)s","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/agentscope/workspace/_k8s/_k8s_workspace.py","lineNumber":342,"sourceCode":"        pvc_name: str,\n        timeout: float = 60.0,\n    ) -> None:\n        \"\"\"Poll until the PVC is fully deleted.\"\"\"\n        from kubernetes_asyncio.client.rest import ApiException\n\n        deadline = asyncio.get_event_loop().time() + timeout\n        while asyncio.get_event_loop().time() < deadline:\n            try:\n                await self._v1.read_namespaced_persistent_volume_claim(\n                    pvc_name,\n                    self._namespace,\n                )\n            except ApiException as e:\n                if e.status == 404:\n                    return\n                raise\n            await asyncio.sleep(1.0)\n        raise RuntimeError(\n            f\"PVC {pvc_name!r} did not finish deleting \" f\"within {timeout}s\",\n        )\n\n    async def _ensure_pod(self) -> None:\n        \"\"\"Create or reuse the workspace Pod.\n\n        Only terminal phases (``Failed``, ``Unknown``, ``Succeeded``)\n        trigger a delete-and-recreate cycle.  ``Pending`` Pods are left\n        for :meth:`_wait_pod_running` which inspects container statuses\n        for early failure detection.\n\n        A Pod whose ``metadata.deletion_timestamp`` is set is already\n        being deleted (by a previous ``close()`` or an external actor)\n        — attaching to it would race the terminator, so we wait for it\n        to disappear and then create a fresh one.\n        \"\"\"\n        from kubernetes_asyncio.client.rest import ApiException\n","sourceCodeStart":324,"sourceCodeEnd":360,"githubUrl":"https://github.com/agentscope-ai/agentscope/blob/e90f1c7592896cc95f6e5ee506194f533378247d/src/agentscope/workspace/_k8s/_k8s_workspace.py#L324-L360","documentation":"Thrown by the K8s workspace when a PersistentVolumeClaim was asked to delete but still exists (not 404) after polling once per second until the timeout. Kubernetes PVC deletion can block on finalizers (e.g. kubernetes.io/pvc-protection) or volumes still attached to a running Pod.","triggerScenarios":"_ensure_pvc recreating the workspace and waiting for the old PVC to disappear; a finalizer stuck because a Pod using the PVC is still terminating; the volume driver (in-tree or CSI) failing to detach/delete the underlying volume.","commonSituations":"Node crash leaving the volume attached; CSI driver bugs or missing driver in clusters where the StorageClass provisions slowly; reclaim policies or third-party finalizers (backup tools) holding the PVC.","solutions":["kubectl get pvc <name> -o yaml — inspect metadata.finalizers and check for pods still mounting it","Force-remove the Pod using the PVC (kubectl delete pod --force) or patch away the pvc-protection finalizer if you know it is safe","Increase the deletion timeout passed to the workspace","Verify the CSI driver for the StorageClass is installed and healthy"],"exampleFix":"# before\nws = K8sWorkspace(...)  # RuntimeError: PVC did not finish deleting within 60s\n\n# after\n# diagnose the stuck finalizer\n# kubectl get pvc ws-agent-123 -o jsonpath='{.metadata.finalizers}'\n# kubectl patch pvc ws-agent-123 -p '{\"metadata\":{\"finalizers\":null}}'\nws = K8sWorkspace(..., pvc_delete_timeout=300.0)","handlingStrategy":"retry","validationCode":"from kubernetes.client import CoreV1Api\nv1 = CoreV1Api()\npvc = v1.read_namespaced_persistent_volume_claim(name, ns)\nstuck = [f for f in pvc.metadata.finalizers or []]  # non-empty => deletion will hang","typeGuard":"null","tryCatchPattern":"try:\n    await ws.start()\nexcept RuntimeError as e:\n    if \"did not finish deleting\" in str(e) and \"PVC\" in str(e):\n        # unstick finalizer, then retry\n        await asyncio.sleep(5)\n        await ws.start()\n    else:\n        raise","preventionTips":["Watch for finalizers on workspace PVCs","Ensure workspace Pods fully terminate before deleting PVCs","Set realistic deletion timeouts on slow storage clusters"],"tags":["kubernetes","pvc","finalizer","timeout","storage"],"backgroundTag":"kubernetes-stuck-deletion","analyzedSha":"e90f1c7592896cc95f6e5ee506194f533378247d","analyzedAt":"2026-08-28T18:24:12.087Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}