apache/beam · error · ValueError

Splittablility incompatible with skipping footers.

Error message

Splittablility incompatible with skipping footers.

What it means

Raised by _TextFileSplitter when constructing a splittable CSV source whose compression or footer-skipping options make dynamic splitting impossible: the splitter cannot both skip footer lines and guarantee splittability, so the incompatible combination is rejected at graph-construction time.

Solutions

Remove skipfooter when splittable=True and pre-clean the file, or filter trailing rows after the read.
Disable splittable=True if skipfooter is essential (smaller files only).
Strip footer lines upstream before writing to storage.

Example fix

// before
read_csv('f.csv', skipfooter=1, splittable=True)
// after
read_csv('f.csv', splittable=True)  # filter footer rows downstream

Defensive patterns

Strategy: validation

Validate before calling

if splittable and kwargs.get('skipfooter', 0):
    raise ValueError('skipfooter incompatible with splittable=True')

Try / catch

try:
    df = read_csv(path, skipfooter=1, splittable=True)
except ValueError:
    df = read_csv(path, splittable=True)  # filter footers downstream

Prevention

Do not combine skipfooter with splittable=True
Clean trailing summary lines upstream
Validate option compatibility in shared csv kwargs

When it happens

Trigger: Calling read_csv(path, skipfooter=1, splittable=True), or a shared kwargs dict containing skipfooter passed to a splittable read.

Common situations: CSV exports with trailing summary/disclaimer lines; configs copied from pandas jobs that used skipfooter=1.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8eab8a35b0e18c09. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/io.py:409

    return str_or_bytes.encode('utf-8')
  else:
    return str_or_bytes


class _TextFileSplitter(_DelimSplitter):
  """Splitter for dynamically sharding CSV files and newline record boundaries.

  Currently does not handle quoted newlines, so is off by default, but such
  support could be added in the future.
  """
  def __init__(self, args, kwargs, read_chunk_size=_DEFAULT_BYTES_CHUNKSIZE):
    if args:
      # TODO(robertwb): Automatically populate kwargs as we do for df methods.
      raise ValueError(
          'Non-path arguments must be passed by keyword '
          'for splittable csv reads.')
    if kwargs.get('skipfooter', 0):
      raise ValueError('Splittablility incompatible with skipping footers.')
    super().__init__(
        _maybe_encode(kwargs.get('lineterminator', b'\n')),
        _DEFAULT_BYTES_CHUNKSIZE)
    self._kwargs = kwargs

  def read_header(self, handle):
    if self._kwargs.get('header', 'infer') == 'infer':
      if 'names' in self._kwargs:
        header = None
      else:
        header = 0
    else:
      header = self._kwargs['header']

    if header is None:
      return self._empty, self._empty

    if isinstance(header, int):

View on GitHub (pinned to 12126d8942)