{"record":{"id":"4f5883056c808fa3","repo":"xai-org/x-algorithm","slug":"error-processing-file-file-e","errorCode":null,"errorMessage":"Error processing file {file}: {e}","messagePattern":"Error processing file (.+?): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/data/parquet_recsys.py","lineNumber":505,"sourceCode":"    def _open_file(self, file: str, pool: ThreadPoolExecutor, active: deque) -> None:\n        rank_logger.info(f\"Worker {self._shard_index}/{self._num_shards} opening file {file}\")\n        path = _resolve_file_path(self._path, file)\n\n        def _impl():\n            try:\n                pf = ParquetFile(path)\n                return LazyRecordBatchIterator(\n                    pf,\n                    self._batch_size,\n                    path,\n                    self._conversion_delay_columns,\n                    self._include_action_delay_columns,\n                )\n            except Exception as e:\n                if \"No such file or directory\" in str(e):\n                    rank_logger.warning(f\"Skipping missing file {file}\")\n                    return None\n                raise ValueError(f\"Error processing file {file}: {e}\") from e\n\n        active.append(pool.submit(_impl))\n\n    @staticmethod\n    def _safe_read(\n        holder: \"LazyRecordBatchIterator\",\n    ) -> pa.RecordBatch | None:\n        try:\n            return holder.read()\n        except StopIteration:\n            return None\n\n    def _drain_files(\n        self,\n        files: list[str],\n        *,\n        pool: ThreadPoolExecutor,\n        prefetched: deque[Future[LazyRecordBatchIterator | None]] | None = None,","sourceCodeStart":487,"sourceCodeEnd":523,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/data/parquet_recsys.py#L487-L523","documentation":"While building the batch pool, _impl() constructs an inner dataset for each file; any exception other than a plain missing-file ('No such file or directory', which is skipped with a warning) is re-raised as ValueError wrapping the original exception with the file name. This is a generic wrapper so the failing file is identifiable in multi-file loads.","triggerScenarios":"Any parquet read error inside per-file dataset construction: corrupted footer, post_sid invariant violation (391), sidecar mismatch, permission error, or bad schema — anything not matching the missing-file message.","commonSituations":"One corrupted file among thousands in an index; Opaque OSError text from storage layers that hides the file context without this wrapper.","solutions":["Read the wrapped exception text after 'Error processing file X' to identify the root cause, then inspect/repair that specific file.","Remove the bad file from the index/metadata and retry if the data is expendable.","If it is a transient storage error, retry the job after the storage issue clears."],"exampleFix":"# before: opaque failure mid-epoch\nError processing file /data/date=.../batch-042.parquet: ...\n\n# after: triage\npq.ParquetFile('/data/date=.../batch-042.parquet').metadata  # reproduce, then repair or drop from index","handlingStrategy":"try-catch","validationCode":"for f in files:\n    if not os.path.isfile(f):\n        continue\n    pq.ParquetFile(f).metadata  # cheap footer probe; surfaces corrupt files early","typeGuard":null,"tryCatchPattern":"try:\n    batch = read_one(file)\nexcept ValueError as e:\n    if 'Error processing file' in str(e):\n        logger.error('quarantining %s', file); dead_letters.append(file); return None\n    raise","preventionTips":["Probe parquet footers for all files at startup and quarantine unreadable ones.","Log the wrapped inner exception to triage root causes quickly."],"tags":["wrapper-exception","parquet","io","diagnostics"],"backgroundTag":"file-read-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}