{"record":{"id":"8c1ef5efa5e28481","repo":"JuliusBrussee/caveman","slug":"recovery-unavailable-8c1ef5","errorCode":"recovery_unavailable","errorMessage":"recovery_unavailable","messagePattern":"recovery_unavailable","errorType":"error_code","errorClass":"MiddlewareError","httpStatus":null,"severity":"error","filePath":"packages/middleware/python/caveman_middleware/crewai.py","lineNumber":71,"sourceCode":"\nclass CavemanRecoveryTool(BaseTool):\n    \"\"\"Executed by CrewAI's existing native tool scheduler.\"\"\"\n    name: str = \"caveman_retrieve\"\n    description: str = RECOVERY_DESCRIPTION\n    args_schema: type[BaseModel] = _RecoveryInput\n    cache_function: Any = _never_cache\n    _binding: Any = PrivateAttr()\n    _model: Any = PrivateAttr()\n\n    def __init__(self, model):\n        super().__init__()\n        self._binding = model.runtime.recovery(model.scope)\n        self._model = weakref.ref(model)\n\n    def _run(self, handle, offset=0, limit=262144, query=\"\"):\n        model = self._model()\n        if model is None or model.closed:\n            raise MiddlewareError(\"recovery_unavailable\")\n        result = self._binding.execute(handle=handle, offset=offset, limit=limit, query=query)\n        return json.dumps(result, ensure_ascii=False, separators=(\",\", \":\"))\n\n\n@dataclass\nclass _Call:\n    model: Any\n    attempt: Attempt\n    finished: bool = False\n    lock: Any = field(default_factory=threading.Lock)\n\n    def finish(self, event, measured=None):\n        with self.lock:\n            if self.finished:\n                return\n            self.finished = True\n        self.attempt.observe(event, measured)\n","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/packages/middleware/python/caveman_middleware/crewai.py#L53-L89","documentation":"_run backs a recovery tool bound to the wrapped model via a weakref. If the CavemanLLM/wrapper has been garbage-collected or closed, the recovery binding can no longer execute, so _run raises MiddlewareError with code recovery_unavailable instead of returning data from self._binding.execute.","triggerScenarios":"Calling the recovery tool's _run after the wrapping model has been closed (model.closed is True) or after the underlying model object was garbage collected (weakref returns None). The binding itself still exists, but its owner does not.","commonSituations":"Long-running crews where the LLM wrapper was closed but the recovery tool outlives it; holding a reference to the recovery tool beyond the model's lifetime; agent teardown happening while a background/async recovery query is still pending.","solutions":["Keep the wrapping model alive for the lifetime of the recovery tool; hold a strong reference while recovery is in use.","Do not close the model until all recovery queries have completed.","Recreate the model (and its recovery binding via runtime.recovery(scope)) and re-run the recovery query.","Handle MiddlewareError code 'recovery_unavailable' by re-establishing the binding instead of retrying _run."],"exampleFix":"// before\ntool._run(handle, offset=0)  # model already closed -> MiddlewareError\n// after\nif not (model is None or model.closed):\n    result = tool._run(handle, offset=0)","handlingStrategy":"try-catch","validationCode":"if model is None or model.closed:\n    # re-bind via runtime.recovery(scope) before calling _run\n    ...","typeGuard":"def recovery_usable(model) -> bool:\n    return model is not None and not model.closed","tryCatchPattern":"try:\n    payload = tool._run(handle, offset=offset)\nexcept MiddlewareError as e:\n    if getattr(e, \"code\", None) == \"recovery_unavailable\":\n        binding = runtime.recovery(scope)  # rebuild and retry once\n    else:\n        raise","preventionTips":["Keep a strong reference to the wrapped model for the tool's lifetime.","Close the model only after all recovery queries finish.","Recreate the recovery binding (runtime.recovery(scope)) on worker restarts.","Check model.closed before issuing recovery calls."],"tags":["crewai","recovery","lifecycle","weakref"],"backgroundTag":"invalid-state-transition","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}