{"record":{"id":"1db5c90f7635d4fd","repo":"xai-org/x-algorithm","slug":"sidecar-sidecar-column-name-has-mat-shape-0","errorCode":null,"errorMessage":"sidecar {sidecar} column {name} has {mat.shape[0]} rows, batch file has {self.num_rows}","messagePattern":"sidecar (.+?) column (.+?) has (.+?) rows, batch file has (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/data/parquet_recsys.py","lineNumber":171,"sourceCode":"        self._sidecar_delays: dict[str, np.ndarray] | None = None\n        self._row_pos: int = 0\n\n    def seek(self):\n        arrow_schema = self.pf.schema_arrow\n        excluded_columns = [\"firstPageSeq\"]\n        valid_columns = [name for name in arrow_schema.names if name not in excluded_columns]\n        self.iter = self.pf.iter_batches(self.batch_size, columns=valid_columns)\n        assert self.iter is not None\n\n        if self.conversion_delay_columns:\n            sidecar = conversion_labels.sidecar_path_for(self.fname)\n            columns = list(self.conversion_delay_columns)\n            if self.include_action_delay_columns:\n                columns += conversion_labels.action_delay_columns(sidecar)\n            self._sidecar_delays = conversion_labels.load_sidecar_delays(sidecar, columns)\n            for name, mat in self._sidecar_delays.items():\n                if mat.shape[0] != self.num_rows:\n                    raise ValueError(\n                        f\"sidecar {sidecar} column {name} has {mat.shape[0]} rows, \"\n                        f\"batch file has {self.num_rows}\"\n                    )\n\n        cnt = 0\n        while cnt < self.rows_to_skip:\n            skipped = next(self.iter)\n            cnt += self.batch_size\n            self._row_pos += skipped.num_rows\n\n    def read(self) -> pa.RecordBatch:\n        if self.rows_to_skip >= self.num_rows:\n            raise StopIteration\n\n        if self.iter is None:\n            self.seek()\n\n        assert self.iter is not None","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/data/parquet_recsys.py#L153-L189","documentation":"During seek(), the loader lazily reads the conversion-label sidecar and verifies each delay column matrix has the same number of rows as the batch parquet file (self.num_rows). A mismatch means the sidecar does not correspond to this exact batch version, so row-aligned delay attachment is impossible.","triggerScenarios":"Batch parquet rewritten (different row count) while the .labels.parquet sidecar stayed from the previous run; pointing sidecar lookup at a partition whose batch was reprocessed; concurrent writer producing a new batch under the old sidecar name.","commonSituations":"Reprocessing pipelines that replace batch files but skip relabeling; partial uploads where the sidecar is from a different shard.","solutions":["Rerun the conversion-labeling job for the affected partition so the sidecar matches the current batch.","Delete the stale sidecar and disable conversion delay columns for that read.","Add a writer-side checksum/row-count pairing so mismatches are caught at write time."],"exampleFix":"# before\n ds.seek(...)  # ValueError: sidecar has 9000 rows, batch has 9500\n\n# after: relabel partition\n rerun_labeling(topic_dir, partition='date=2026-08-28')\n ds.seek(...)","handlingStrategy":"fallback","validationCode":"sc = pq.ParquetFile(sidecar).metadata.num_rows\nbc = pq.ParquetFile(batch_file).metadata.num_rows\nif sc != bc:\n    logger.warning('stale sidecar %s (%d vs %d)', sidecar, sc, bc)","typeGuard":null,"tryCatchPattern":"try:\n    ds.seek(pos)\nexcept ValueError as e:\n    if 'sidecar' in str(e) and 'rows' in str(e):\n        disable_conversion_columns_and_retry(ds)\n    else:\n        raise","preventionTips":["Pair batch and sidecar writes atomically and embed row counts in sidecar metadata.","Relabel partitions after any batch rewrite."],"tags":["row-count-mismatch","sidecar","conversion-labels","data-integrity"],"backgroundTag":"data-shape-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}