{"record":{"id":"cd6174253c03572f","repo":"xai-org/x-algorithm","slug":"post-sid-schema-invariant-violated-in-file-path","errorCode":null,"errorMessage":"post_sid schema invariant violated in {file_path}: expected {n} rows × {sid_num_levels} codes = {expected_total} flat ints, got flat_values.size={flat_values.size}, offsets span {int(offsets[-1] - offsets[0]) if offsets.size else 0}","messagePattern":"post_sid schema invariant violated in (.+?): expected (.+?) rows × (.+?) codes = (.+?) flat ints, got flat_values\\.size=(.+?), offsets span (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"phoenix/xrex/data/parquet_recsys.py","lineNumber":112,"sourceCode":"    author_ids = np.asarray(\n        table.column(\"author_id\").to_numpy(zero_copy_only=False), dtype=np.uint64\n    )\n    post_creation_datetimes = None\n    if read_creation_datetime:\n        created_at = table.column(\"created_at\").to_numpy(zero_copy_only=False)\n        post_creation_datetimes = np.asarray(created_at, dtype=\"datetime64[ms]\")\n\n    post_sids: np.ndarray | None = None\n    if read_post_sid:\n        n = len(table)\n        sid_col = table.column(\"post_sid\").combine_chunks()\n        offsets = sid_col.offsets.to_numpy(zero_copy_only=False)\n        flat_values = sid_col.values.to_numpy(zero_copy_only=False)\n        expected_total = n * sid_num_levels\n        if n > 0 and (\n            flat_values.size != expected_total or offsets[-1] - offsets[0] != expected_total\n        ):\n            raise ValueError(\n                f\"post_sid schema invariant violated in {file_path}: \"\n                f\"expected {n} rows × {sid_num_levels} codes = {expected_total} flat ints, \"\n                f\"got flat_values.size={flat_values.size}, offsets span \"\n                f\"{int(offsets[-1] - offsets[0]) if offsets.size else 0}\"\n            )\n        post_sids = np.ascontiguousarray(flat_values.reshape(-1, sid_num_levels), dtype=np.int32)\n        n_with = int((post_sids[:, 0] != -1).sum()) if n > 0 else 0\n        rank_logger.info(\n            f\"Packed post_sid for {n_with:,}/{n:,} rows ({n_with / max(n, 1):.1%}) into [{n}, {sid_num_levels}] int32\"\n        )\n\n    if len(post_ids) == 0 or len(author_ids) == 0 or len(post_ids) != len(author_ids):\n        rank_logger.info(\n            f\"Global ids file is empty or has mismatched post and author ids or creation datetimes: {file_path}\"\n        )\n        return None, None, None, None\n\n    rank_logger.info(","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/data/parquet_recsys.py#L94-L130","documentation":"load_global_ids_from_parquet_file() expects the post_sid column to be a ListArray where n rows each contain exactly sid_num_levels int codes — i.e. flat_values.size == n * sid_num_levels and the offsets span equals the same. Any deviation means the schema invariant (fixed-width sid per row) is broken, and it raises ValueError with the file, expected total, actual flat size, and offsets span.","triggerScenarios":"A parquet file whose post_sid rows have variable code counts (e.g. some rows empty, some with 3 codes when sid_num_levels=2); corrupted or truncated file; a writer bug emitting ragged sid lists.","commonSituations":"Data written by an older producer with a different sid arity; partially flushed files from a crashed job; upstream join producing duplicate/missing sid parts.","solutions":["Inspect the file: pq.ParquetFile(path).read()['post_sid'] and check per-row list lengths.","Exclude/repair the offending file (rewrite with fixed sid_num_levels per row) or re-run the producer for that batch.","If the arity legitimately changed, update sid_num_levels consistently across writer and reader."],"exampleFix":"// before: ragged rows, e.g. [[1,2],[3],[4,5,6]]\nload_global_ids_from_parquet_file(f, sid_num_levels=2)  # ValueError\n\n// after: normalized rows [[1,2],[3,0],[4,5]] (or regenerate file)\nload_global_ids_from_parquet_file(f_fixed, sid_num_levels=2)","handlingStrategy":"try-catch","validationCode":"sid = pq.read_table(path, columns=['post_sid'])['post_sid'].combine_chunks()\noff = sid.offsets.to_numpy()\nlevels = off[1:] - off[:-1]\nif not (levels == levels[0]).all():\n    print(f'{path}: ragged post_sid, skip/repair')","typeGuard":null,"tryCatchPattern":"try:\n    ids = load_global_ids_from_parquet_file(path, sid_num_levels)\nexcept ValueError as e:\n    logger.error('quarantining %s: %s', path, e)\n    quarantine.append(path); return None","preventionTips":["Run a nightly invariant check over new parquet files before admitting them to the index.","Have writers emit fixed-size sid lists (pad) rather than variable-length lists."],"tags":["parquet","schema-invariant","sid","data-integrity"],"backgroundTag":"schema-validation-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}