xai-org/x-algorithm · error · ValueError

sidecar {sidecar} column {name} has {mat.shape[0]} rows, bat

Error message

sidecar {sidecar} column {name} has {mat.shape[0]} rows, batch file has {self.num_rows}

What it means

During seek(), the loader lazily reads the conversion-label sidecar and verifies each delay column matrix has the same number of rows as the batch parquet file (self.num_rows). A mismatch means the sidecar does not correspond to this exact batch version, so row-aligned delay attachment is impossible.

Source

Thrown at phoenix/xrex/data/parquet_recsys.py:171

        self._sidecar_delays: dict[str, np.ndarray] | None = None
        self._row_pos: int = 0

    def seek(self):
        arrow_schema = self.pf.schema_arrow
        excluded_columns = ["firstPageSeq"]
        valid_columns = [name for name in arrow_schema.names if name not in excluded_columns]
        self.iter = self.pf.iter_batches(self.batch_size, columns=valid_columns)
        assert self.iter is not None

        if self.conversion_delay_columns:
            sidecar = conversion_labels.sidecar_path_for(self.fname)
            columns = list(self.conversion_delay_columns)
            if self.include_action_delay_columns:
                columns += conversion_labels.action_delay_columns(sidecar)
            self._sidecar_delays = conversion_labels.load_sidecar_delays(sidecar, columns)
            for name, mat in self._sidecar_delays.items():
                if mat.shape[0] != self.num_rows:
                    raise ValueError(
                        f"sidecar {sidecar} column {name} has {mat.shape[0]} rows, "
                        f"batch file has {self.num_rows}"
                    )

        cnt = 0
        while cnt < self.rows_to_skip:
            skipped = next(self.iter)
            cnt += self.batch_size
            self._row_pos += skipped.num_rows

    def read(self) -> pa.RecordBatch:
        if self.rows_to_skip >= self.num_rows:
            raise StopIteration

        if self.iter is None:
            self.seek()

        assert self.iter is not None

View on GitHub (pinned to 24c60942c5)

Solutions

1. Rerun the conversion-labeling job for the affected partition so the sidecar matches the current batch.
2. Delete the stale sidecar and disable conversion delay columns for that read.
3. Add a writer-side checksum/row-count pairing so mismatches are caught at write time.

Example fix

# before
 ds.seek(...)  # ValueError: sidecar has 9000 rows, batch has 9500

# after: relabel partition
 rerun_labeling(topic_dir, partition='date=2026-08-28')
 ds.seek(...)

sc = pq.ParquetFile(sidecar).metadata.num_rows
bc = pq.ParquetFile(batch_file).metadata.num_rows
if sc != bc:
    logger.warning('stale sidecar %s (%d vs %d)', sidecar, sc, bc)

try:
    ds.seek(pos)
except ValueError as e:
    if 'sidecar' in str(e) and 'rows' in str(e):
        disable_conversion_columns_and_retry(ds)
    else:
        raise

When it happens

Trigger: Batch parquet rewritten (different row count) while the .labels.parquet sidecar stayed from the previous run; pointing sidecar lookup at a partition whose batch was reprocessed; concurrent writer producing a new batch under the old sidecar name.

Common situations: Reprocessing pipelines that replace batch files but skip relabeling; partial uploads where the sidecar is from a different shard.

Related errors

• {name}: delay rows {mat.shape[0]} != batch rows {batch.num_r
• not a partition batch file path: {batch_file_path}
• not a parquet path: {batch_file_path}
• sidecar {sidecar_path} missing column(s) {missing}; availabl
• batch missing {ACTION_MULTIHOT_COLUMN}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/1db5c90f7635d4fd. Report an issue: GitHub.