xai-org/x-algorithm · error · ValueError

not a parquet path: {batch_file_path}

Error message

not a parquet path: {batch_file_path}

What it means

After locating the partition segment in the batch path, sidecar_path_for() requires the relative remainder to end with '.parquet' so it can derive the '.labels.parquet' sidecar name. A path whose filename lacks the .parquet extension cannot be mapped to a sidecar, so ValueError is raised. This is a weaker check that catches directories, temp files, and non-parquet inputs.

Source

Thrown at phoenix/xrex/data/conversion_labels.py:48

        (n for n in names if n.startswith(ACTION_DELAY_PREFIX)),
        key=lambda n: int(n[len(ACTION_DELAY_PREFIX) :]),
    )


def action_index_of(column: str) -> int:
    if not column.startswith(ACTION_DELAY_PREFIX):
        raise ValueError(f"not an action delay column: {column}")
    return int(column[len(ACTION_DELAY_PREFIX) :])


def sidecar_path_for(batch_file_path: str) -> str:
    m = _PARTITION_SEG_RE.search(batch_file_path)
    if m is None:
        raise ValueError(f"not a partition batch file path: {batch_file_path}")
    root = batch_file_path[: m.start()]
    rel = batch_file_path[m.start() :]
    if not rel.endswith(".parquet"):
        raise ValueError(f"not a parquet path: {batch_file_path}")
    return os.path.join(root, "labels", rel[: -len(".parquet")] + ".labels.parquet")


def load_sidecar_delays(
    sidecar_path: str, columns: list[str] | None = None
) -> dict[str, np.ndarray]:
    columns = columns or [DELAY_COLUMN]
    schema_names = pq.ParquetFile(sidecar_path).schema_arrow.names
    missing = [c for c in columns if c not in schema_names]
    if missing:
        raise ValueError(
            f"sidecar {sidecar_path} missing column(s) {missing}; available: {schema_names}"
        )
    t = pq.read_table(sidecar_path, columns=columns)
    out: dict[str, np.ndarray] = {}
    for name in columns:
        col = t.column(name).combine_chunks()
        if isinstance(col, pa.ChunkedArray):

View on GitHub (pinned to 24c60942c5)

Solutions

1. Pass the final, fully-written .parquet batch file path.
2. Filter index/metadata listings to only '*.parquet' entries.
3. Wait for writers to finish renaming temp files before the loader discovers them.

Example fix

// before
sidecar_path_for('/d/date=2026-08-28/hour=13/batch-0007.tmp')  # ValueError

// after
sidecar_path_for('/d/date=2026-08-28/hour=13/batch-0007.parquet')

def usable_batch_path(p: str) -> bool:
    return p.endswith('.parquet') and PARTITION_SEG.search(p) is not None

def is_parquet_partition_file(p: str) -> bool:
    return p.endswith('.parquet') and bool(PARTITION_SEG.search(p))

try:
    sidecar = sidecar_path_for(p)
except ValueError:
    continue  # skip temp/non-parquet entries

When it happens

Trigger: Calling sidecar_path_for('.../date=2026-08-28/hour=13/batch-0007.snappy') or with a '.csv', '.arrow', partially-written '.parquet.tmp', or a directory path ending in a partition segment.

Common situations: Index files listing temporary or in-progress files that were later renamed; hand-built paths with wrong extensions; passing a partition directory instead of a file.

Related errors

• not a partition batch file path: {batch_file_path}
• sidecar {sidecar_path} missing column(s) {missing}; availabl
• no cached cls parquet under {split_dir}
• {name}: delay rows {mat.shape[0]} != batch rows {batch.num_r
• batch missing {ACTION_MULTIHOT_COLUMN}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/bc97c37c83a9feae. Report an issue: GitHub.