{"record":{"id":"5a26c82ff9207513","repo":"xai-org/x-algorithm","slug":"sidecar-sidecar-path-missing-column-s-missing","errorCode":null,"errorMessage":"sidecar {sidecar_path} missing column(s) {missing}; available: {schema_names}","messagePattern":"sidecar (.+?) missing column\\(s\\) (.+?); available: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/data/conversion_labels.py","lineNumber":59,"sourceCode":"def sidecar_path_for(batch_file_path: str) -> str:\n    m = _PARTITION_SEG_RE.search(batch_file_path)\n    if m is None:\n        raise ValueError(f\"not a partition batch file path: {batch_file_path}\")\n    root = batch_file_path[: m.start()]\n    rel = batch_file_path[m.start() :]\n    if not rel.endswith(\".parquet\"):\n        raise ValueError(f\"not a parquet path: {batch_file_path}\")\n    return os.path.join(root, \"labels\", rel[: -len(\".parquet\")] + \".labels.parquet\")\n\n\ndef load_sidecar_delays(\n    sidecar_path: str, columns: list[str] | None = None\n) -> dict[str, np.ndarray]:\n    columns = columns or [DELAY_COLUMN]\n    schema_names = pq.ParquetFile(sidecar_path).schema_arrow.names\n    missing = [c for c in columns if c not in schema_names]\n    if missing:\n        raise ValueError(\n            f\"sidecar {sidecar_path} missing column(s) {missing}; available: {schema_names}\"\n        )\n    t = pq.read_table(sidecar_path, columns=columns)\n    out: dict[str, np.ndarray] = {}\n    for name in columns:\n        col = t.column(name).combine_chunks()\n        if isinstance(col, pa.ChunkedArray):\n            col = col.chunk(0)\n        seq_len = col.type.list_size\n        flat = col.flatten().to_numpy(zero_copy_only=False).astype(np.int64)\n        out[name] = flat.reshape(len(col), seq_len)\n    return out\n\n\ndef attach_delays(batch: pa.RecordBatch, delays: dict[str, np.ndarray]) -> pa.RecordBatch:\n    for name, mat in delays.items():\n        if mat.shape[0] != batch.num_rows:\n            raise ValueError(f\"{name}: delay rows {mat.shape[0]} != batch rows {batch.num_rows}\")","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/data/conversion_labels.py#L41-L77","documentation":"load_sidecar_delays() opens the .labels.parquet sidecar, reads its Arrow schema names, and verifies every requested delay column exists before reading. If any requested column (default: the single DELAY_COLUMN) is absent, it raises ValueError listing the missing columns and the sidecar's actual schema. It means the sidecar was written by an older/newer pipeline version or with a different column set.","triggerScenarios":"Setting conversion_delay_columns=['conv_delay_v2'] when the sidecar only contains the default delay column; enabling include_action_delay_columns when the sidecar predates per-action delay columns; pointing at a sidecar generated by a different labeling job.","commonSituations":"Schema drift after a pipeline upgrade renames or drops delay columns; reusing old sidecar files against new configs; sidecars written with columns=None default only.","solutions":["Inspect the sidecar schema with pq.ParquetFile(sidecar).schema_arrow.names and align conversion_delay_columns with what exists.","Regenerate the sidecars with the current labeling pipeline so they contain the required columns.","Drop the requested column from config if the data genuinely lacks it."],"exampleFix":"// before\ndelays = load_sidecar_delays(s, columns=['action_delay_17'])  # ValueError\n\n// after\nnames = pq.ParquetFile(s).schema_arrow.names\ndelays = load_sidecar_delays(s, [c for c in wanted if c in names])","handlingStrategy":"validation","validationCode":"import pyarrow.parquet as pq\nnames = set(pq.ParquetFile(sidecar).schema_arrow.names)\nwanted = [c for c in requested_columns if c in names]","typeGuard":null,"tryCatchPattern":"try:\n    delays = load_sidecar_delays(sidecar, cols)\nexcept ValueError as e:\n    logger.error('sidecar schema mismatch %s: %s', sidecar, e)\n    raise","preventionTips":["Log sidecar schema at job start and compare against configured columns.","Version sidecar schemas with the pipeline so drift is detectable."],"tags":["parquet","schema","conversion-labels","validation"],"backgroundTag":"schema-validation-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}