apache/beam · error · RateLimitExceeded

Rate Limit Exceeded, Could not process this batch.

Error message

Rate Limit Exceeded, Could not process this batch.

What it means

RemoteModelHandler.run_inference consults the configured RateLimiter before dispatching a batch. If the limiter's allow(hits_added=len(batch)) says the batch would exceed the request quota, RateLimitExceeded is raised for the whole batch instead of sending it.

Source

Thrown at sdks/python/apache_beam/ml/inference/base.py:515

    Args:
      batch: A sequence of examples or features.
      model: The model used to make inferences.
      inference_args: Extra arguments for models whose inference call requires
        extra parameters.

    Returns:
      An Iterable of Predictions.
    """
    if self._rate_limiter:
      if self._shared_rate_limiter is None:

        def init_limiter():
          return self._rate_limiter

        self._shared_rate_limiter = self._shared_handle.acquire(init_limiter)

      if not self._shared_rate_limiter.allow(hits_added=len(batch)):
        raise RateLimitExceeded(
            "Rate Limit Exceeded, "
            "Could not process this batch.")

    self.throttler.throttle()

    try:
      req_time = time.time()
      predictions = self.request(batch, model, inference_args)
      self.throttler.successful_request(req_time * _MILLISECOND_TO_SECOND)
      return predictions
    except Exception as e:
      self.logger.error("exception raised as part of request, got %s", e)
      raise

  @abstractmethod
  def request(
      self,
      batch: Sequence[ExampleT],

View on GitHub (pinned to 12126d8942)

Solutions

1. Reduce batch_size in RunInference so len(batch) fits within the rate limiter's allowance.
2. Configure a RateLimiter (e.g. with higher max_per_second) matching your provider quota.
3. Enable automatic retries: wrap in retry configuration so RateLimitExceeded is retried after backoff (the handler's retry_on_exception supports this).
4. Request a quota increase from the model provider or upgrade the API plan.

Example fix

# before
handler = OpenAIEmbeddingsHandler(api_key=key, rate_limiter=RateLimiter(max_per_second=1))
# with batch_size=100 -> exceeds allowance

# after
result = pcoll | RunInference(handler.with_batch_size(10))

# ensure batch fits under limiter allowance
max_batch = rate_limiter.max_per_second * window_seconds
assert batch_size <= max_batch, f'batch_size {batch_size} exceeds rate allowance {max_batch}'

def batch_within_limit(limiter, batch):
    return len(batch) <= getattr(limiter, 'max_per_second', 1)

from apache_beam.ml.inference.base import RateLimitExceeded
try:
    result = pcoll | RunInference(handler)
except RateLimitExceeded:
    # rerun with smaller batch size or higher rate limit
    handler = handler.with_batch_size(max(1, batch_size // 2))

When it happens

Trigger: Calling run_inference on a RemoteModelHandler constructed with a rate_limiter whose remaining allowance is less than len(batch); large batches against a tight per-minute/per-second quota.

Common situations: Using remote embedding/chat APIs (e.g. OpenAI) with strict RPM/TPM quotas; batch size set too large for the plan's limits; many parallel Beam workers sharing a quota via the shared rate limiter.

Related errors

• Cannot override RemoteModelHandler.load_model, implement cre
• Cannot override RemoteModelHandler.run_inference, implement
• Cannot make make an unkeyed model handler with pre or postpr
• Cannot use an unkeyed model handler with pre or postprocessi
• Empty list maps to model handler {mh}. All model handlers mu

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/11d29f9d4f6335da. Report an issue: GitHub.