xai-org/x-algorithm · error · ValueError

sidecar {sidecar_path} missing column(s) {missing}; availabl

Error message

sidecar {sidecar_path} missing column(s) {missing}; available: {schema_names}

What it means

load_sidecar_delays() opens the .labels.parquet sidecar, reads its Arrow schema names, and verifies every requested delay column exists before reading. If any requested column (default: the single DELAY_COLUMN) is absent, it raises ValueError listing the missing columns and the sidecar's actual schema. It means the sidecar was written by an older/newer pipeline version or with a different column set.

Source

Thrown at phoenix/xrex/data/conversion_labels.py:59

def sidecar_path_for(batch_file_path: str) -> str:
    m = _PARTITION_SEG_RE.search(batch_file_path)
    if m is None:
        raise ValueError(f"not a partition batch file path: {batch_file_path}")
    root = batch_file_path[: m.start()]
    rel = batch_file_path[m.start() :]
    if not rel.endswith(".parquet"):
        raise ValueError(f"not a parquet path: {batch_file_path}")
    return os.path.join(root, "labels", rel[: -len(".parquet")] + ".labels.parquet")


def load_sidecar_delays(
    sidecar_path: str, columns: list[str] | None = None
) -> dict[str, np.ndarray]:
    columns = columns or [DELAY_COLUMN]
    schema_names = pq.ParquetFile(sidecar_path).schema_arrow.names
    missing = [c for c in columns if c not in schema_names]
    if missing:
        raise ValueError(
            f"sidecar {sidecar_path} missing column(s) {missing}; available: {schema_names}"
        )
    t = pq.read_table(sidecar_path, columns=columns)
    out: dict[str, np.ndarray] = {}
    for name in columns:
        col = t.column(name).combine_chunks()
        if isinstance(col, pa.ChunkedArray):
            col = col.chunk(0)
        seq_len = col.type.list_size
        flat = col.flatten().to_numpy(zero_copy_only=False).astype(np.int64)
        out[name] = flat.reshape(len(col), seq_len)
    return out


def attach_delays(batch: pa.RecordBatch, delays: dict[str, np.ndarray]) -> pa.RecordBatch:
    for name, mat in delays.items():
        if mat.shape[0] != batch.num_rows:
            raise ValueError(f"{name}: delay rows {mat.shape[0]} != batch rows {batch.num_rows}")

View on GitHub (pinned to 24c60942c5)

Solutions

1. Inspect the sidecar schema with pq.ParquetFile(sidecar).schema_arrow.names and align conversion_delay_columns with what exists.
2. Regenerate the sidecars with the current labeling pipeline so they contain the required columns.
3. Drop the requested column from config if the data genuinely lacks it.

Example fix

// before
delays = load_sidecar_delays(s, columns=['action_delay_17'])  # ValueError

// after
names = pq.ParquetFile(s).schema_arrow.names
delays = load_sidecar_delays(s, [c for c in wanted if c in names])

import pyarrow.parquet as pq
names = set(pq.ParquetFile(sidecar).schema_arrow.names)
wanted = [c for c in requested_columns if c in names]

try:
    delays = load_sidecar_delays(sidecar, cols)
except ValueError as e:
    logger.error('sidecar schema mismatch %s: %s', sidecar, e)
    raise

When it happens

Trigger: Setting conversion_delay_columns=['conv_delay_v2'] when the sidecar only contains the default delay column; enabling include_action_delay_columns when the sidecar predates per-action delay columns; pointing at a sidecar generated by a different labeling job.

Common situations: Schema drift after a pipeline upgrade renames or drops delay columns; reusing old sidecar files against new configs; sidecars written with columns=None default only.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• batch missing {ACTION_MULTIHOT_COLUMN}
• not a partition batch file path: {batch_file_path}
• not a parquet path: {batch_file_path}
• batch missing newEventMask; conversion-label folding is cand
• type checking expression %s failed: invalid argument type: %

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/5a26c82ff9207513. Report an issue: GitHub.