xai-org/x-algorithm · critical · ValueError

post_sid schema invariant violated in {file_path}: expected

Error message

post_sid schema invariant violated in {file_path}: expected {n} rows × {sid_num_levels} codes = {expected_total} flat ints, got flat_values.size={flat_values.size}, offsets span {int(offsets[-1] - offsets[0]) if offsets.size else 0}

What it means

load_global_ids_from_parquet_file() expects the post_sid column to be a ListArray where n rows each contain exactly sid_num_levels int codes — i.e. flat_values.size == n * sid_num_levels and the offsets span equals the same. Any deviation means the schema invariant (fixed-width sid per row) is broken, and it raises ValueError with the file, expected total, actual flat size, and offsets span.

Source

Thrown at phoenix/xrex/data/parquet_recsys.py:112

    author_ids = np.asarray(
        table.column("author_id").to_numpy(zero_copy_only=False), dtype=np.uint64
    )
    post_creation_datetimes = None
    if read_creation_datetime:
        created_at = table.column("created_at").to_numpy(zero_copy_only=False)
        post_creation_datetimes = np.asarray(created_at, dtype="datetime64[ms]")

    post_sids: np.ndarray | None = None
    if read_post_sid:
        n = len(table)
        sid_col = table.column("post_sid").combine_chunks()
        offsets = sid_col.offsets.to_numpy(zero_copy_only=False)
        flat_values = sid_col.values.to_numpy(zero_copy_only=False)
        expected_total = n * sid_num_levels
        if n > 0 and (
            flat_values.size != expected_total or offsets[-1] - offsets[0] != expected_total
        ):
            raise ValueError(
                f"post_sid schema invariant violated in {file_path}: "
                f"expected {n} rows × {sid_num_levels} codes = {expected_total} flat ints, "
                f"got flat_values.size={flat_values.size}, offsets span "
                f"{int(offsets[-1] - offsets[0]) if offsets.size else 0}"
            )
        post_sids = np.ascontiguousarray(flat_values.reshape(-1, sid_num_levels), dtype=np.int32)
        n_with = int((post_sids[:, 0] != -1).sum()) if n > 0 else 0
        rank_logger.info(
            f"Packed post_sid for {n_with:,}/{n:,} rows ({n_with / max(n, 1):.1%}) into [{n}, {sid_num_levels}] int32"
        )

    if len(post_ids) == 0 or len(author_ids) == 0 or len(post_ids) != len(author_ids):
        rank_logger.info(
            f"Global ids file is empty or has mismatched post and author ids or creation datetimes: {file_path}"
        )
        return None, None, None, None

    rank_logger.info(

View on GitHub (pinned to 24c60942c5)

Solutions

1. Inspect the file: pq.ParquetFile(path).read()['post_sid'] and check per-row list lengths.
2. Exclude/repair the offending file (rewrite with fixed sid_num_levels per row) or re-run the producer for that batch.
3. If the arity legitimately changed, update sid_num_levels consistently across writer and reader.

Example fix

// before: ragged rows, e.g. [[1,2],[3],[4,5,6]]
load_global_ids_from_parquet_file(f, sid_num_levels=2)  # ValueError

// after: normalized rows [[1,2],[3,0],[4,5]] (or regenerate file)
load_global_ids_from_parquet_file(f_fixed, sid_num_levels=2)

sid = pq.read_table(path, columns=['post_sid'])['post_sid'].combine_chunks()
off = sid.offsets.to_numpy()
levels = off[1:] - off[:-1]
if not (levels == levels[0]).all():
    print(f'{path}: ragged post_sid, skip/repair')

try:
    ids = load_global_ids_from_parquet_file(path, sid_num_levels)
except ValueError as e:
    logger.error('quarantining %s: %s', path, e)
    quarantine.append(path); return None

When it happens

Trigger: A parquet file whose post_sid rows have variable code counts (e.g. some rows empty, some with 3 codes when sid_num_levels=2); corrupted or truncated file; a writer bug emitting ragged sid lists.

Common situations: Data written by an older producer with a different sid arity; partially flushed files from a crashed job; upstream join producing duplicate/missing sid parts.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• no cached cls parquet under {split_dir}
• Unknown {dataset_type=} for the SID retrieval family
• not a parquet path: {batch_file_path}
• sidecar {sidecar_path} missing column(s) {missing}; availabl
• {name}: delay rows {mat.shape[0]} != batch rows {batch.num_r

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/cd6174253c03572f. Report an issue: GitHub.