{"record":{"id":"66e7de314caba4ad","repo":"apache/beam","slug":"ngrams-separator-must-be-specified-when-ngram-range-is-not-1","errorCode":null,"errorMessage":"ngrams_separator must be specified when ngram_range is not (1, 1)","messagePattern":"ngrams_separator must be specified when ngram_range is not \\(1, 1\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/transforms/tft.py","lineNumber":595,"sourceCode":"    set of consecutive n-grams.\n\n    Args:\n      columns: A list of column names to apply the transformation on.\n      split_string_by_delimiter: (Optional) A string that specifies the\n        delimiter to split the input strings before computing ngrams.\n      ngram_range: A tuple of integers(inclusive) specifying the range of\n        n-gram sizes.\n      ngrams_separator: A string that will be inserted between each ngram.\n      name: A name for the operation (optional).\n    \"\"\"\n    super().__init__(columns)\n    self.ngram_range = ngram_range\n    self.ngrams_separator = ngrams_separator\n    self.name = name\n    self.split_string_by_delimiter = split_string_by_delimiter\n\n    if ngram_range != (1, 1) and not ngrams_separator:\n      raise ValueError(\n          'ngrams_separator must be specified when ngram_range is not (1, 1)')\n\n  def apply_transform(\n      self, data: common_types.TensorType,\n      output_column_name: str) -> dict[str, common_types.TensorType]:\n    if self.split_string_by_delimiter:\n      data = self._split_string_with_delimiter(\n          data, self.split_string_by_delimiter)\n    output = tft.ngrams(data, self.ngram_range, self.ngrams_separator)\n    return {output_column_name: output}\n\n\n@register_input_dtype(str)\nclass BagOfWords(TFTOperation):\n  def __init__(\n      self,\n      columns: list[str],\n      split_string_by_delimiter: Optional[str] = None,","sourceCodeStart":577,"sourceCodeEnd":613,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/transforms/tft.py#L577-L613","documentation":"An ngram-producing TFT op validates that when ngram_range is anything other than (1, 1), a ngrams_separator must be provided, because tft.ngrams needs a delimiter to join tokens into ngrams. __init__ raises a ValueError when ngrams are requested without a separator.","triggerScenarios":"Constructing the ngram transform (e.g. NGrams-style op around tft.ngrams) with ngram_range=(2, 3) (or any non-(1,1)) and ngrams_separator=None or empty string.","commonSituations":"Using the default ngram_range=(1,1) then widening it to (1,2) or (2,2) without updating ngrams_separator; copy-pasted config that omits the separator key.","solutions":["Pass ngrams_separator, e.g. ngrams_separator=' ' for space-joined tokens or another delimiter appropriate to the data.","Keep ngram_range=(1, 1) if unigrams only are needed, in which case no separator is required.","Ensure text is tokenized consistently so the chosen separator matches the tokenization."],"exampleFix":"# before\ntft.NGrams(columns=['text'], ngram_range=(2, 2))  # separator missing\n\n# after\ntft.NGrams(columns=['text'], ngram_range=(2, 2), ngrams_separator=' ')","handlingStrategy":"validation","validationCode":"def make_ngram_op(columns, ngram_range, ngrams_separator=None):\n    if ngram_range != (1, 1) and not ngrams_separator:\n        raise ValueError('ngrams_separator required when ngram_range != (1, 1)')\n    return tft.NGrams(columns=columns, ngram_range=ngram_range, ngrams_separator=ngrams_separator)","typeGuard":"def ngram_config_valid(ngram_range, ngrams_separator) -> bool:\n    return ngram_range == (1, 1) or bool(ngrams_separator)","tryCatchPattern":"try:\n    op = tft.NGrams(columns=['text'], ngram_range=(2, 2), ngrams_separator=sep)\nexcept ValueError as e:\n    if 'ngrams_separator' in str(e):\n        op = tft.NGrams(columns=['text'], ngram_range=(1, 1))  # fall back to unigrams\n    else:\n        raise","preventionTips":["Pair every widened ngram_range with an explicit ngrams_separator in configs.","Validate ngram configs at load time (range vs separator).","Keep token delimiters consistent between tokenization and ngrams_separator.","Default ngram_range to (1, 1) unless ngrams are truly needed."],"tags":["python","apache-beam","tft","ngrams"],"backgroundTag":"missing-required-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}