{"record":{"id":"411e101d0cc554d1","repo":"apache/beam","slug":"vocab-size-is-not-specified-tried-to-infer-vocab-size-from","errorCode":null,"errorMessage":"vocab_size is not specified. Tried to infer vocab_size from the input data using tft.get_num_buckets_for_transformed_feature, but failed. Please specify vocab_size explicitly.","messagePattern":"vocab_size is not specified\\. Tried to infer vocab_size from the input data using tft\\.get_num_buckets_for_transformed_feature, but failed\\. Please specify vocab_size explicitly\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/transforms/tft.py","lineNumber":506,"sourceCode":"    super().__init__(columns)\n    self.vocab_size = vocab_size\n    self.smooth = smooth\n    self.name = name\n    self.tfidf_weight = None\n\n  def apply_transform(\n      self, data: common_types.TensorType,\n      output_column_name: str) -> common_types.TensorType:\n\n    if self.vocab_size is None:\n      try:\n        _LOGGER.info(\n            'vocab_size is not specified. Trying to infer vocab_size '\n            'from the input data using '\n            'tft.get_num_buckets_for_transformed_feature.')\n        vocab_size = tft.get_num_buckets_for_transformed_feature(data)\n      except RuntimeError:\n        raise RuntimeError(\n            'vocab_size is not specified. Tried to infer vocab_size from the '\n            'input data using tft.get_num_buckets_for_transformed_feature, but '\n            'failed. Please specify vocab_size explicitly.')\n    else:\n      vocab_size = self.vocab_size\n\n    vocab_index, tfidf_weight = tft.tfidf(\n      data,\n      vocab_size,\n      self.smooth,\n      self.name\n    )\n\n    output = {\n        output_column_name + '_vocab_index': vocab_index,\n        output_column_name + '_tfidf_weight': tfidf_weight\n    }\n    return output","sourceCodeStart":488,"sourceCodeEnd":524,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/transforms/tft.py#L488-L524","documentation":"When vocab_size is omitted in a vocab/bucketizing TFT op, the code attempts to infer it at transform-apply time via tft.get_num_buckets_for_transformed_feature(data). If that inference itself raises RuntimeError, the code re-raises a descriptive RuntimeError telling the user to specify vocab_size explicitly, since inference from already-transformed data failed.","triggerScenarios":"Constructing an op like ComputeAndApplyVocab or HashAndScale without vocab_size, where the pipeline's data does not allow get_num_buckets_for_transformed_feature to compute a bucket count (e.g. data already transformed, or feature not present in the analytics dataset).","commonSituations":"Inferring vocab size on a read-artifact (inference) pass where the transformed feature metadata isn't available, or chaining ops so the column is no longer raw when vocab_size is needed.","solutions":["Set vocab_size explicitly in the op constructor, e.g. ComputeAndApplyVocab(columns=['text'], vocab_size=10000).","If inference is desired, ensure the write-artifact analytics pass runs first and the op is applied on raw (untransformed) data.","Estimate the vocabulary from the source dataset offline and pass that value in."],"exampleFix":"# before\nop = tft.ComputeAndApplyVocab(columns=['words'])  # vocab_size inferred, fails\n\n# after\nop = tft.ComputeAndApplyVocab(columns=['words'], vocab_size=20000)","handlingStrategy":"validation","validationCode":"def make_vocab_op(columns, vocab_size=None):\n    if vocab_size is None:\n        # inference can fail mid-pipeline; require it up front for reliability\n        raise ValueError('Specify vocab_size explicitly to avoid inference failure')\n    return tft.ComputeAndApplyVocab(columns=columns, vocab_size=vocab_size)","typeGuard":"def has_vocab_size(op) -> bool:\n    return getattr(op, 'vocab_size', None) is not None","tryCatchPattern":"try:\n    result = pcoll | MLTransform(transforms).with_write_artifact_location(loc)\nexcept RuntimeError as e:\n    if 'vocab_size' in str(e):\n        # rebuild configs with explicit vocab_size\n        transforms = [rebuild_with_vocab_size(t, default_vocab_size) for t in transforms]\n        result = pcoll | MLTransform(transforms).with_write_artifact_location(loc)\n    else:\n        raise","preventionTips":["Always set vocab_size explicitly for vocab/bucketizing ops.","Only rely on inference in the write-artifact pass on raw data.","Estimate vocab size offline from the source dataset.","Never chain transforms that mutate a column before a vocab op needing its raw form."],"tags":["python","apache-beam","tft","vocab-size"],"backgroundTag":"missing-required-config-field","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}