xai-org/x-algorithm · error · ValueError

not a partition batch file path: {batch_file_path}

Error message

not a partition batch file path: {batch_file_path}

What it means

sidecar_path_for() computes the conversion-label sidecar path for a partition batch file by regex-matching a Hive-style partition segment (e.g. 'date=.../hour=...') in the path. If the regex _PARTITION_SEG_RE finds no partition segment, the path cannot be relocated into the 'labels/' sibling directory, so the function raises ValueError. This almost always means the caller passed a file path that was not produced under the expected partitioned topic layout.

Source

Thrown at phoenix/xrex/data/conversion_labels.py:44

def action_delay_columns(sidecar_path: str) -> list[str]:
    names = pq.ParquetFile(sidecar_path).schema_arrow.names
    return sorted(
        (n for n in names if n.startswith(ACTION_DELAY_PREFIX)),
        key=lambda n: int(n[len(ACTION_DELAY_PREFIX) :]),
    )


def action_index_of(column: str) -> int:
    if not column.startswith(ACTION_DELAY_PREFIX):
        raise ValueError(f"not an action delay column: {column}")
    return int(column[len(ACTION_DELAY_PREFIX) :])


def sidecar_path_for(batch_file_path: str) -> str:
    m = _PARTITION_SEG_RE.search(batch_file_path)
    if m is None:
        raise ValueError(f"not a partition batch file path: {batch_file_path}")
    root = batch_file_path[: m.start()]
    rel = batch_file_path[m.start() :]
    if not rel.endswith(".parquet"):
        raise ValueError(f"not a parquet path: {batch_file_path}")
    return os.path.join(root, "labels", rel[: -len(".parquet")] + ".labels.parquet")


def load_sidecar_delays(
    sidecar_path: str, columns: list[str] | None = None
) -> dict[str, np.ndarray]:
    columns = columns or [DELAY_COLUMN]
    schema_names = pq.ParquetFile(sidecar_path).schema_arrow.names
    missing = [c for c in columns if c not in schema_names]
    if missing:
        raise ValueError(
            f"sidecar {sidecar_path} missing column(s) {missing}; available: {schema_names}"
        )
    t = pq.read_table(sidecar_path, columns=columns)

View on GitHub (pinned to 24c60942c5)

Solutions

1. Ensure batch file paths passed to the dataset contain a Hive partition segment such as 'date=2026-08-28/hour=13/' before the .parquet filename.
2. Regenerate or fix the index file / .valid_batches.json so every listed path includes the partition directories.
3. If your data genuinely has no partitions, skip conversion-label features (do not set include_action_delay_columns / conversion_delay_columns) so sidecar_path_for is never called.

Example fix

// before
path = '/data/batch-0007.parquet'
sidecar = sidecar_path_for(path)  # ValueError

// after
path = '/data/date=2026-08-28/hour=13/batch-0007.parquet'
sidecar = sidecar_path_for(path)  # '/data/labels/date=2026-08-28/hour=13/batch-0007.labels.parquet'

import re
PARTITION_SEG = re.compile(r'[^/]+=[^/]+/')
def is_partition_batch_path(p: str) -> bool:
    return PARTITION_SEG.search(p) is not None and p.endswith('.parquet')

def is_partition_batch_path(p: str) -> bool:
    return bool(PARTITION_SEG.search(p)) and p.endswith('.parquet')

try:
    sidecar = sidecar_path_for(p)
except ValueError as e:
    logger.warning('skipping non-partition path %s: %s', p, e)
    sidecar = None

When it happens

Trigger: Calling seek() or sidecar_path_for() directly with a path like '/data/batch-42.parquet' that lacks any 'key=value/' partition directory; passing an index_path entry pointing at flat, non-Hive-partitioned files; passing a directory or URL instead of a partition file path.

Common situations: Pointing the dataset at a directory tree written before Hive partitioning was adopted; mixing index files that list relative, un-partitioned paths; typos in topic_dir that strip the partition component.

Related errors

• not a parquet path: {batch_file_path}
• sidecar {sidecar_path} missing column(s) {missing}; availabl
• batch missing {ACTION_MULTIHOT_COLUMN}
• type checking expression %s failed: invalid argument type: %
• Non-optional parameter %s must be declared before optional p

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/516f515c0f9ea7f0. Report an issue: GitHub.