apache/beam · error · RuntimeError

Timeout waiting to acquire model: {tag} after {wait_time_ela

Error message

Timeout waiting to acquire model: {tag} after {wait_time_elapsed:.1f} seconds.

What it means

The keyed model manager's acquire_model waits in a queue for its turn to load the model; if the wait exceeds _wait_timeout_seconds, it raises RuntimeError naming the model tag and elapsed time. This prevents workers from blocking forever when the model holder never releases or loads.

Source

Thrown at sdks/python/apache_beam/ml/inference/model_manager.py:522

      # by ticket number as FIFO.
      self.logging_info(
          "Acquire Queued: tag=%s, priority=%d "
          "total models count=%s ticket num=%s",
          tag,
          current_priority,
          len(self._models[tag]),
          ticket_num)
      heapq.heappush(self._wait_queue, my_ticket)

      est_cost = 0.0
      is_unknown = False
      wait_time_start = time.time()

      try:
        while True:
          wait_time_elapsed = time.time() - wait_time_start
          if wait_time_elapsed > self._wait_timeout_seconds:
            raise RuntimeError(
                f"Timeout waiting to acquire model: {tag} "
                f"after {wait_time_elapsed:.1f} seconds.")
          if not self._wait_queue or self._wait_queue[
              0].ticket_num != ticket_num:
            self.logging_info(
                "Waiting for its turn: tag=%s ticket num=%s", tag, ticket_num)
            self._wait_in_queue(my_ticket)
            continue

          # Re-evaluate priority in case model became known during wait
          is_unknown = self._estimator.is_unknown(tag)
          real_priority = 0 if is_unknown else 1

          # If priority changed, reinsert into queue and wait
          if current_priority != real_priority:
            heapq.heappop(self._wait_queue)
            current_priority = real_priority
            my_ticket = QueueTicket(current_priority, ticket_num, tag)

View on GitHub (pinned to 12126d8942)

Solutions

1. Increase _wait_timeout_seconds to exceed worst-case model load time.
2. Verify the holder releases the model (release_model) in all code paths, including exceptions.
3. Reduce contention: fewer workers per model tag, or increase model resources.
4. Retry the acquisition; transient contention is often the cause.

Example fix

// before
handler = KeyedModelHandler(...); manager.wait_timeout_seconds = 30  # too short for a 5-min load
// after
KeyedModelHandler(..., wait_timeout_seconds=600)  # exceeds max load time

assert wait_timeout_seconds >= expected_model_load_seconds, 'timeout too small for model load'

def timeout_is_sane(manager) -> bool:
    return manager._wait_timeout_seconds > 0

for attempt in range(3):
    try:
        model = manager.acquire_model(tag)
        break
    except RuntimeError as e:
        if 'Timeout waiting to acquire model' in str(e) and attempt < 2:
            time.sleep(5 * (attempt + 1))
        else:
            raise

When it happens

Trigger: Worker calls acquire_model(tag) while another worker holds the model for longer than _wait_timeout_seconds (heavy load, slow model loading, a crashed holder).

Common situations: Streaming pipelines with many workers sharing one model; too-short timeout relative to model load time; a worker that acquired the model and died without releasing it.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Encountered exceptions in finalize_write: %s
• PubSub publish timeout exceeded {self.FLUSH_TIMEOUT_SECS} se
• Timeout {timeout} exceeded while completing request: {reques
• 'timeout' must be a non-negative number
• Cannot schedule new tasks after thread pool has been shutdow

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bec6b993947d9f7f. Report an issue: GitHub.