apache/beam · error · RuntimeError

vocab_size is not specified. Tried to infer vocab_size from…

Error message

vocab_size is not specified. Tried to infer vocab_size from the input data using tft.get_num_buckets_for_transformed_feature, but failed. Please specify vocab_size explicitly.

What it means

When vocab_size is omitted in a vocab/bucketizing TFT op, the code attempts to infer it at transform-apply time via tft.get_num_buckets_for_transformed_feature(data). If that inference itself raises RuntimeError, the code re-raises a descriptive RuntimeError telling the user to specify vocab_size explicitly, since inference from already-transformed data failed.

Solutions

Set vocab_size explicitly in the op constructor, e.g. ComputeAndApplyVocab(columns=['text'], vocab_size=10000).
If inference is desired, ensure the write-artifact analytics pass runs first and the op is applied on raw (untransformed) data.
Estimate the vocabulary from the source dataset offline and pass that value in.

Example fix

# before
op = tft.ComputeAndApplyVocab(columns=['words'])  # vocab_size inferred, fails

# after
op = tft.ComputeAndApplyVocab(columns=['words'], vocab_size=20000)

Defensive patterns

Strategy: validation

Validate before calling

def make_vocab_op(columns, vocab_size=None):
    if vocab_size is None:
        # inference can fail mid-pipeline; require it up front for reliability
        raise ValueError('Specify vocab_size explicitly to avoid inference failure')
    return tft.ComputeAndApplyVocab(columns=columns, vocab_size=vocab_size)

Type guard

def has_vocab_size(op) -> bool:
    return getattr(op, 'vocab_size', None) is not None

Try / catch

try:
    result = pcoll | MLTransform(transforms).with_write_artifact_location(loc)
except RuntimeError as e:
    if 'vocab_size' in str(e):
        # rebuild configs with explicit vocab_size
        transforms = [rebuild_with_vocab_size(t, default_vocab_size) for t in transforms]
        result = pcoll | MLTransform(transforms).with_write_artifact_location(loc)
    else:
        raise

Prevention

Always set vocab_size explicitly for vocab/bucketizing ops.
Only rely on inference in the write-artifact pass on raw data.
Estimate vocab size offline from the source dataset.
Never chain transforms that mutate a column before a vocab op needing its raw form.

When it happens

Trigger: Constructing an op like ComputeAndApplyVocab or HashAndScale without vocab_size, where the pipeline's data does not allow get_num_buckets_for_transformed_feature to compute a bucket count (e.g. data already transformed, or feature not present in the analytics dataset).

Common situations: Inferring vocab size on a read-artifact (inference) pass where the transformed feature metadata isn't available, or chaining ops so the column is no longer raw when vocab_size is needed.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/411e101d0cc554d1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/transforms/tft.py:506

    super().__init__(columns)
    self.vocab_size = vocab_size
    self.smooth = smooth
    self.name = name
    self.tfidf_weight = None

  def apply_transform(
      self, data: common_types.TensorType,
      output_column_name: str) -> common_types.TensorType:

    if self.vocab_size is None:
      try:
        _LOGGER.info(
            'vocab_size is not specified. Trying to infer vocab_size '
            'from the input data using '
            'tft.get_num_buckets_for_transformed_feature.')
        vocab_size = tft.get_num_buckets_for_transformed_feature(data)
      except RuntimeError:
        raise RuntimeError(
            'vocab_size is not specified. Tried to infer vocab_size from the '
            'input data using tft.get_num_buckets_for_transformed_feature, but '
            'failed. Please specify vocab_size explicitly.')
    else:
      vocab_size = self.vocab_size

    vocab_index, tfidf_weight = tft.tfidf(
      data,
      vocab_size,
      self.smooth,
      self.name
    )

    output = {
        output_column_name + '_vocab_index': vocab_index,
        output_column_name + '_tfidf_weight': tfidf_weight
    }
    return output

View on GitHub (pinned to 12126d8942)