xai-org/x-algorithm · error · ValueError

Index file {index_path} not found

Error message

Index file {index_path} not found

What it means

In index mode, _get_ready_batches_from_index() requires the configured index file to exist on disk; if index_path is None (shouldn't happen after __init__ validation) or os.path.isfile fails, it raises 'Index file {index_path} not found'. This surfaces at data-loading time rather than construction because the file may be created later.

Source

Thrown at phoenix/xrex/data/parquet_recsys.py:432

        if start > max_batch:
            return []

        ready: list[list[str]] = []
        for bid in range(start, max_batch + 1):
            my_files = [
                _batch_path(self._topic_dir, p, bid)
                for p in range(num_partitions)
                if p % self._num_shards == self._shard_index
            ]
            ready.append(my_files)
            self._next_batch_id = bid + 1

        return ready

    def _get_ready_batches_from_index(self) -> list[list[str]]:
        index_path = self._index_path
        if index_path is None or not os.path.isfile(index_path):
            raise ValueError(f"Index file {index_path} not found")

        with open(index_path) as f:
            all_files = [line.strip() for line in f if line.strip()]

        if self._date_range is not None:
            start_str, end_str = self._date_range
            start_date = (
                datetime.strptime(start_str, DATE_TIME_FORMAT)
                if start_str.lower() != "none"
                else None
            )
            end_date = (
                datetime.strptime(end_str, DATE_TIME_FORMAT) if end_str.lower() != "none" else None
            )
            if start_date is not None or end_date is not None:
                filtered: list[str] = []
                for file in all_files:
                    try:

View on GitHub (pinned to 24c60942c5)

Solutions

1. Create the index file (one parquet path per line) at the configured path before starting the loader.
2. Fix mounts/permissions so the path is visible, or correct the path in config.
3. Retry/wait for the index-producing step if it runs concurrently.

Example fix

# before: files.txt missing
ds = ParquetRecsysDataset(..., index_path='/data/files.txt')
next(iter(ds))  # ValueError: not found

# after
open('/data/files.txt','w').write('\n'.join(paths))
next(iter(ds))

import os
if index_path and not os.path.isfile(index_path):
    build_index_file(topic_dir, index_path)  # write one path per line

for attempt in range(5):
    try:
        batches = ds._get_ready_batches()
        break
    except ValueError as e:
        if 'not found' in str(e) and attempt < 4:
            time.sleep(30); continue
        raise

When it happens

Trigger: Starting training before the index file was generated; index file on an unmounted NFS volume; typo in index_path; file deleted between config time and read time.

Common situations: Race between a job that builds the index and the trainer that consumes it; environment where the data volume mount name changed.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Did not find any files matching {file_pattern}
• no cached cls parquet under {split_dir}
• Keytab file not found: {self.config.keytab_path}
• enable_stale_post is unsupported on grpc_recsys, which does
• Uknown {dataset_type=}, must be one of {DATASET_TYPES}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/71e43516d68a1f9e. Report an issue: GitHub.