apache/beam · error · ValueError

Max batch_size exceeded. Batch size needs to be smaller than

Error message

Max batch_size exceeded. Batch size needs to be smaller than {}

What it means

AnnotateImage.__init__ validates max_batch_size against AnnotateImage.MAX_BATCH_SIZE (the Cloud Vision API limit on features per AnnotateImageRequest); a larger requested batch cannot be sent as a single request and is rejected up front.

Solutions

  1. Pass max_batch_size <= AnnotateImage.MAX_BATCH_SIZE (or omit it to use the default)
  2. If you need more throughput, scale horizontally (more workers) rather than raising batch size
Defensive patterns

Strategy: validation

When it happens

Trigger: Thrown at sdks/python/apache_beam/ml/gcp/visionml.py:130 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bb916ab86370402b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/gcp/visionml.py:130

          context_side_input =
            (
              p
              | "Image contexts" >> beam.Create(image_contexts)
            )

          visionml.AnnotateImage(features,
            context_side_input=beam.pvalue.AsDict(context_side_input)))
      metadata: (Optional[Sequence[Tuple[str, str]]]): Optional.
        Additional metadata that is provided to the method.
    """
    super().__init__()
    self.features = features
    self.retry = retry
    self.timeout = timeout
    self.max_batch_size = max_batch_size or AnnotateImage.MAX_BATCH_SIZE
    if self.max_batch_size > AnnotateImage.MAX_BATCH_SIZE:
      raise ValueError(
          'Max batch_size exceeded. '
          'Batch size needs to be smaller than {}'.format(
              AnnotateImage.MAX_BATCH_SIZE))
    self.min_batch_size = min_batch_size or AnnotateImage.MIN_BATCH_SIZE
    self.client_options = client_options
    self.context_side_input = context_side_input
    self.metadata = metadata

  def expand(self, pvalue):
    return (
        pvalue
        | FlatMap(self._create_image_annotation_pairs, self.context_side_input)
        | util.BatchElements(
            min_batch_size=self.min_batch_size,
            max_batch_size=self.max_batch_size)
        | ParDo(
            _ImageAnnotateFn(
                features=self.features,

View on GitHub (pinned to 12126d8942)