apache/beam · error · ValueError

ngrams_separator must be specified when ngram_range is not…

Error message

ngrams_separator must be specified when ngram_range is not (1, 1)

What it means

An ngram-producing TFT op validates that when ngram_range is anything other than (1, 1), a ngrams_separator must be provided, because tft.ngrams needs a delimiter to join tokens into ngrams. __init__ raises a ValueError when ngrams are requested without a separator.

Solutions

Pass ngrams_separator, e.g. ngrams_separator=' ' for space-joined tokens or another delimiter appropriate to the data.
Keep ngram_range=(1, 1) if unigrams only are needed, in which case no separator is required.
Ensure text is tokenized consistently so the chosen separator matches the tokenization.

Example fix

# before
tft.NGrams(columns=['text'], ngram_range=(2, 2))  # separator missing

# after
tft.NGrams(columns=['text'], ngram_range=(2, 2), ngrams_separator=' ')

Defensive patterns

Strategy: validation

Validate before calling

def make_ngram_op(columns, ngram_range, ngrams_separator=None):
    if ngram_range != (1, 1) and not ngrams_separator:
        raise ValueError('ngrams_separator required when ngram_range != (1, 1)')
    return tft.NGrams(columns=columns, ngram_range=ngram_range, ngrams_separator=ngrams_separator)

Type guard

def ngram_config_valid(ngram_range, ngrams_separator) -> bool:
    return ngram_range == (1, 1) or bool(ngrams_separator)

Try / catch

try:
    op = tft.NGrams(columns=['text'], ngram_range=(2, 2), ngrams_separator=sep)
except ValueError as e:
    if 'ngrams_separator' in str(e):
        op = tft.NGrams(columns=['text'], ngram_range=(1, 1))  # fall back to unigrams
    else:
        raise

Prevention

Pair every widened ngram_range with an explicit ngrams_separator in configs.
Validate ngram configs at load time (range vs separator).
Keep token delimiters consistent between tokenization and ngrams_separator.
Default ngram_range to (1, 1) unless ngrams are truly needed.

When it happens

Trigger: Constructing the ngram transform (e.g. NGrams-style op around tft.ngrams) with ngram_range=(2, 3) (or any non-(1,1)) and ngrams_separator=None or empty string.

Common situations: Using the default ngram_range=(1,1) then widening it to (1,2) or (2,2) without updating ngrams_separator; copy-pasted config that omits the separator key.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/66e7de314caba4ad. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/transforms/tft.py:595

    set of consecutive n-grams.

    Args:
      columns: A list of column names to apply the transformation on.
      split_string_by_delimiter: (Optional) A string that specifies the
        delimiter to split the input strings before computing ngrams.
      ngram_range: A tuple of integers(inclusive) specifying the range of
        n-gram sizes.
      ngrams_separator: A string that will be inserted between each ngram.
      name: A name for the operation (optional).
    """
    super().__init__(columns)
    self.ngram_range = ngram_range
    self.ngrams_separator = ngrams_separator
    self.name = name
    self.split_string_by_delimiter = split_string_by_delimiter

    if ngram_range != (1, 1) and not ngrams_separator:
      raise ValueError(
          'ngrams_separator must be specified when ngram_range is not (1, 1)')

  def apply_transform(
      self, data: common_types.TensorType,
      output_column_name: str) -> dict[str, common_types.TensorType]:
    if self.split_string_by_delimiter:
      data = self._split_string_with_delimiter(
          data, self.split_string_by_delimiter)
    output = tft.ngrams(data, self.ngram_range, self.ngrams_separator)
    return {output_column_name: output}


@register_input_dtype(str)
class BagOfWords(TFTOperation):
  def __init__(
      self,
      columns: list[str],
      split_string_by_delimiter: Optional[str] = None,

View on GitHub (pinned to 12126d8942)