{"record":{"id":"efb4b326165b62a6","repo":"apache/beam","slug":"delimiter-must-not-self-overlap","errorCode":null,"errorMessage":"Delimiter must not self-overlap.","messagePattern":"Delimiter must not self-overlap\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/textio.py","lineNumber":166,"sourceCode":"\n    self._strip_trailing_newlines = strip_trailing_newlines\n    self._compression_type = compression_type\n    self._coder = coder\n    self._buffer_size = buffer_size\n    if skip_header_lines < 0:\n      raise ValueError(\n          'Cannot skip negative number of header lines: %d' % skip_header_lines)\n    elif skip_header_lines > 10:\n      _LOGGER.warning(\n          'Skipping %d header lines. Skipping large number of header '\n          'lines might significantly slow down processing.')\n    self._skip_header_lines = skip_header_lines\n    self._header_matcher, self._header_processor = header_processor_fns\n    if delimiter is not None:\n      if not isinstance(delimiter, bytes) or len(delimiter) == 0:\n        raise ValueError('Delimiter must be a non-empty bytes sequence.')\n      if self._is_self_overlapping(delimiter):\n        raise ValueError('Delimiter must not self-overlap.')\n    self._delimiter = delimiter\n    if escapechar is not None:\n      if not (isinstance(escapechar, bytes) and len(escapechar) == 1):\n        raise ValueError(\n            \"escapechar must be bytes of size 1: '%s'\" % escapechar)\n    self._escapechar = escapechar\n\n  def display_data(self):\n    parent_dd = super().display_data()\n    parent_dd['strip_newline'] = DisplayDataItem(\n        self._strip_trailing_newlines, label='Strip Trailing New Lines')\n    parent_dd['buffer_size'] = DisplayDataItem(\n        self._buffer_size, label='Buffer Size')\n    parent_dd['coder'] = DisplayDataItem(self._coder.__class__, label='Coder')\n    return parent_dd\n\n  def read_records(self, file_name, range_tracker):\n    start_offset = range_tracker.start_position()","sourceCodeStart":148,"sourceCodeEnd":184,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/textio.py#L148-L184","documentation":"_TextSource rejects delimiters that self-overlap because the binary record-splitting algorithm cannot correctly split streams with such delimiters. It raises ValueError at construction when _is_self_overlapping(delimiter) is true, i.e. the delimiter has a proper prefix equal to a suffix (like b'aa' or b'abab').","triggerScenarios":"Constructing ReadFromText (or CSV/JSON transforms using _TextSource) with delimiter=b'aa', b'aba', b'\\n\\n' or any pattern where one occurrence overlaps another.","commonSituations":"Using repeated-character separators like b';;' or b'---' as record delimiters when the user wants multi-character splitting.","solutions":["Choose a self-overlap-free delimiter, e.g. single bytes or patterns with distinct first/last bytes","Split manually with a DoFn/beam.Map using str.split on the overlapping pattern","Combine a safe delimiter with post-processing to emulate the overlapping pattern"],"exampleFix":"// before\nReadFromText('gs://bucket/f', delimiter=b'aa')\n// after\nReadFromText('gs://bucket/f', delimiter=b'a')  # or a non-self-overlapping sequence","handlingStrategy":"validation","validationCode":"def self_overlaps(d: bytes) -> bool:\n    return any(d.startswith(d[k+1:]) and len(d[k+1:]) > 0 for k in range(len(d)-1))\n# call before constructing: assert not self_overlaps(delim)","typeGuard":"def is_non_self_overlapping(d) -> bool:\n    return d is None or (isinstance(d, bytes) and len(d) > 0 and d[0] != d[-1] or len(d) == 1)","tryCatchPattern":"try:\n    src = ReadFromText(path, delimiter=delim)\nexcept ValueError as e:\n    if 'self-overlap' in str(e):\n        src = ReadFromText(path, delimiter=delim[:1])  # fall back to first byte\n    else:\n        raise","preventionTips":["Prefer single-byte or clearly non-overlapping multi-byte delimiters","Check that first and last bytes of the delimiter differ","Pre-split overlapping patterns in a DoFn instead of at source level"],"tags":["python","beam","io","validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}