apache/beam · error · ValueError

Delimiter must not self-overlap.

Error message

Delimiter must not self-overlap.

What it means

_TextSource rejects delimiters that self-overlap because the binary record-splitting algorithm cannot correctly split streams with such delimiters. It raises ValueError at construction when _is_self_overlapping(delimiter) is true, i.e. the delimiter has a proper prefix equal to a suffix (like b'aa' or b'abab').

Source

Thrown at sdks/python/apache_beam/io/textio.py:166

    self._strip_trailing_newlines = strip_trailing_newlines
    self._compression_type = compression_type
    self._coder = coder
    self._buffer_size = buffer_size
    if skip_header_lines < 0:
      raise ValueError(
          'Cannot skip negative number of header lines: %d' % skip_header_lines)
    elif skip_header_lines > 10:
      _LOGGER.warning(
          'Skipping %d header lines. Skipping large number of header '
          'lines might significantly slow down processing.')
    self._skip_header_lines = skip_header_lines
    self._header_matcher, self._header_processor = header_processor_fns
    if delimiter is not None:
      if not isinstance(delimiter, bytes) or len(delimiter) == 0:
        raise ValueError('Delimiter must be a non-empty bytes sequence.')
      if self._is_self_overlapping(delimiter):
        raise ValueError('Delimiter must not self-overlap.')
    self._delimiter = delimiter
    if escapechar is not None:
      if not (isinstance(escapechar, bytes) and len(escapechar) == 1):
        raise ValueError(
            "escapechar must be bytes of size 1: '%s'" % escapechar)
    self._escapechar = escapechar

  def display_data(self):
    parent_dd = super().display_data()
    parent_dd['strip_newline'] = DisplayDataItem(
        self._strip_trailing_newlines, label='Strip Trailing New Lines')
    parent_dd['buffer_size'] = DisplayDataItem(
        self._buffer_size, label='Buffer Size')
    parent_dd['coder'] = DisplayDataItem(self._coder.__class__, label='Coder')
    return parent_dd

  def read_records(self, file_name, range_tracker):
    start_offset = range_tracker.start_position()

View on GitHub (pinned to 12126d8942)

Solutions

1. Choose a self-overlap-free delimiter, e.g. single bytes or patterns with distinct first/last bytes
2. Split manually with a DoFn/beam.Map using str.split on the overlapping pattern
3. Combine a safe delimiter with post-processing to emulate the overlapping pattern

Example fix

// before
ReadFromText('gs://bucket/f', delimiter=b'aa')
// after
ReadFromText('gs://bucket/f', delimiter=b'a')  # or a non-self-overlapping sequence

def self_overlaps(d: bytes) -> bool:
    return any(d.startswith(d[k+1:]) and len(d[k+1:]) > 0 for k in range(len(d)-1))
# call before constructing: assert not self_overlaps(delim)

def is_non_self_overlapping(d) -> bool:
    return d is None or (isinstance(d, bytes) and len(d) > 0 and d[0] != d[-1] or len(d) == 1)

try:
    src = ReadFromText(path, delimiter=delim)
except ValueError as e:
    if 'self-overlap' in str(e):
        src = ReadFromText(path, delimiter=delim[:1])  # fall back to first byte
    else:
        raise

When it happens

Trigger: Constructing ReadFromText (or CSV/JSON transforms using _TextSource) with delimiter=b'aa', b'aba', b'\n\n' or any pattern where one occurrence overlaps another.

Common situations: Using repeated-character separators like b';;' or b'---' as record delimiters when the user wants multi-character splitting.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Delimiter must be a non-empty bytes sequence.
• escapechar must be bytes of size 1: '%s'
• MatchContinuously interval must be positive.
• Please install apache_beam[dataframe]
• The --setup_file option expects the full path to a file name

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/efb4b326165b62a6. Report an issue: GitHub.