{"record":{"id":"bec6b993947d9f7f","repo":"apache/beam","slug":"timeout-waiting-to-acquire-model-tag-after-wait-time-elapsed","errorCode":null,"errorMessage":"Timeout waiting to acquire model: {tag} after {wait_time_elapsed:.1f} seconds.","messagePattern":"Timeout waiting to acquire model: (.+?) after (.+?) seconds\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/inference/model_manager.py","lineNumber":522,"sourceCode":"      # by ticket number as FIFO.\n      self.logging_info(\n          \"Acquire Queued: tag=%s, priority=%d \"\n          \"total models count=%s ticket num=%s\",\n          tag,\n          current_priority,\n          len(self._models[tag]),\n          ticket_num)\n      heapq.heappush(self._wait_queue, my_ticket)\n\n      est_cost = 0.0\n      is_unknown = False\n      wait_time_start = time.time()\n\n      try:\n        while True:\n          wait_time_elapsed = time.time() - wait_time_start\n          if wait_time_elapsed > self._wait_timeout_seconds:\n            raise RuntimeError(\n                f\"Timeout waiting to acquire model: {tag} \"\n                f\"after {wait_time_elapsed:.1f} seconds.\")\n          if not self._wait_queue or self._wait_queue[\n              0].ticket_num != ticket_num:\n            self.logging_info(\n                \"Waiting for its turn: tag=%s ticket num=%s\", tag, ticket_num)\n            self._wait_in_queue(my_ticket)\n            continue\n\n          # Re-evaluate priority in case model became known during wait\n          is_unknown = self._estimator.is_unknown(tag)\n          real_priority = 0 if is_unknown else 1\n\n          # If priority changed, reinsert into queue and wait\n          if current_priority != real_priority:\n            heapq.heappop(self._wait_queue)\n            current_priority = real_priority\n            my_ticket = QueueTicket(current_priority, ticket_num, tag)","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/inference/model_manager.py#L504-L540","documentation":"The keyed model manager's acquire_model waits in a queue for its turn to load the model; if the wait exceeds _wait_timeout_seconds, it raises RuntimeError naming the model tag and elapsed time. This prevents workers from blocking forever when the model holder never releases or loads.","triggerScenarios":"Worker calls acquire_model(tag) while another worker holds the model for longer than _wait_timeout_seconds (heavy load, slow model loading, a crashed holder).","commonSituations":"Streaming pipelines with many workers sharing one model; too-short timeout relative to model load time; a worker that acquired the model and died without releasing it.","solutions":["Increase _wait_timeout_seconds to exceed worst-case model load time.","Verify the holder releases the model (release_model) in all code paths, including exceptions.","Reduce contention: fewer workers per model tag, or increase model resources.","Retry the acquisition; transient contention is often the cause."],"exampleFix":"// before\nhandler = KeyedModelHandler(...); manager.wait_timeout_seconds = 30  # too short for a 5-min load\n// after\nKeyedModelHandler(..., wait_timeout_seconds=600)  # exceeds max load time","handlingStrategy":"retry","validationCode":"assert wait_timeout_seconds >= expected_model_load_seconds, 'timeout too small for model load'","typeGuard":"def timeout_is_sane(manager) -> bool:\n    return manager._wait_timeout_seconds > 0","tryCatchPattern":"for attempt in range(3):\n    try:\n        model = manager.acquire_model(tag)\n        break\n    except RuntimeError as e:\n        if 'Timeout waiting to acquire model' in str(e) and attempt < 2:\n            time.sleep(5 * (attempt + 1))\n        else:\n            raise","preventionTips":["Set the wait timeout well above worst-case model load time","Ensure holders release the model in finally blocks","Monitor holder health; a dead holder causes permanent timeouts"],"tags":["python","apache-beam","timeout","model-manager","concurrency"],"backgroundTag":"request-timeout","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}