{"record":{"id":"8eab8a35b0e18c09","repo":"apache/beam","slug":"splittablility-incompatible-with-skipping-footers","errorCode":null,"errorMessage":"Splittablility incompatible with skipping footers.","messagePattern":"Splittablility incompatible with skipping footers\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/io.py","lineNumber":409,"sourceCode":"    return str_or_bytes.encode('utf-8')\n  else:\n    return str_or_bytes\n\n\nclass _TextFileSplitter(_DelimSplitter):\n  \"\"\"Splitter for dynamically sharding CSV files and newline record boundaries.\n\n  Currently does not handle quoted newlines, so is off by default, but such\n  support could be added in the future.\n  \"\"\"\n  def __init__(self, args, kwargs, read_chunk_size=_DEFAULT_BYTES_CHUNKSIZE):\n    if args:\n      # TODO(robertwb): Automatically populate kwargs as we do for df methods.\n      raise ValueError(\n          'Non-path arguments must be passed by keyword '\n          'for splittable csv reads.')\n    if kwargs.get('skipfooter', 0):\n      raise ValueError('Splittablility incompatible with skipping footers.')\n    super().__init__(\n        _maybe_encode(kwargs.get('lineterminator', b'\\n')),\n        _DEFAULT_BYTES_CHUNKSIZE)\n    self._kwargs = kwargs\n\n  def read_header(self, handle):\n    if self._kwargs.get('header', 'infer') == 'infer':\n      if 'names' in self._kwargs:\n        header = None\n      else:\n        header = 0\n    else:\n      header = self._kwargs['header']\n\n    if header is None:\n      return self._empty, self._empty\n\n    if isinstance(header, int):","sourceCodeStart":391,"sourceCodeEnd":427,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/io.py#L391-L427","documentation":"Raised by _TextFileSplitter when constructing a splittable CSV source whose compression or footer-skipping options make dynamic splitting impossible: the splitter cannot both skip footer lines and guarantee splittability, so the incompatible combination is rejected at graph-construction time.","triggerScenarios":"Calling read_csv(path, skipfooter=1, splittable=True), or a shared kwargs dict containing skipfooter passed to a splittable read.","commonSituations":"CSV exports with trailing summary/disclaimer lines; configs copied from pandas jobs that used skipfooter=1.","solutions":["Remove skipfooter when splittable=True and pre-clean the file, or filter trailing rows after the read.","Disable splittable=True if skipfooter is essential (smaller files only).","Strip footer lines upstream before writing to storage."],"exampleFix":"// before\nread_csv('f.csv', skipfooter=1, splittable=True)\n// after\nread_csv('f.csv', splittable=True)  # filter footer rows downstream","handlingStrategy":"validation","validationCode":"if splittable and kwargs.get('skipfooter', 0):\n    raise ValueError('skipfooter incompatible with splittable=True')","typeGuard":null,"tryCatchPattern":"try:\n    df = read_csv(path, skipfooter=1, splittable=True)\nexcept ValueError:\n    df = read_csv(path, splittable=True)  # filter footers downstream","preventionTips":["Do not combine skipfooter with splittable=True","Clean trailing summary lines upstream","Validate option compatibility in shared csv kwargs"],"tags":["python","apache-beam","csv","splittable"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}