xai-org/x-algorithm · error · ValueError

Index file not found

Error message

Index file {index_path} not found

What it means

In index mode, _get_ready_batches_from_index() requires the configured index file to exist on disk; if index_path is None (shouldn't happen after __init__ validation) or os.path.isfile fails, it raises 'Index file {index_path} not found'. This surfaces at data-loading time rather than construction because the file may be created later.

Solutions

1. Create the index file (one parquet path per line) at the configured path before starting the loader.
2. Fix mounts/permissions so the path is visible, or correct the path in config.
3. Retry/wait for the index-producing step if it runs concurrently.

Example fix

# before: files.txt missing
ds = ParquetRecsysDataset(..., index_path='/data/files.txt')
next(iter(ds))  # ValueError: not found

# after
open('/data/files.txt','w').write('\n'.join(paths))
next(iter(ds))

import os
if index_path and not os.path.isfile(index_path):
    build_index_file(topic_dir, index_path)  # write one path per line

for attempt in range(5):
    try:
        batches = ds._get_ready_batches()
        break
    except ValueError as e:
        if 'not found' in str(e) and attempt < 4:
            time.sleep(30); continue
        raise

When it happens

Trigger: Starting training before the index file was generated; index file on an unmounted NFS volume; typo in index_path; file deleted between config time and read time.

Common situations: Race between a job that builds the index and the trainer that consumes it; environment where the data volume mount name changed.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Did not find any files matching
• no cached cls parquet under
• Cannot load metadata from
• Compiler annotation file
• Dataset '$dataset' does not match $datasetRegex.

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/71e43516d68a1f9e. Report an issue: GitHub.

Appendix: source

Thrown at phoenix/xrex/data/parquet_recsys.py:432

        if start > max_batch:
            return []

        ready: list[list[str]] = []
        for bid in range(start, max_batch + 1):
            my_files = [
                _batch_path(self._topic_dir, p, bid)
                for p in range(num_partitions)
                if p % self._num_shards == self._shard_index
            ]
            ready.append(my_files)
            self._next_batch_id = bid + 1

        return ready

    def _get_ready_batches_from_index(self) -> list[list[str]]:
        index_path = self._index_path
        if index_path is None or not os.path.isfile(index_path):
            raise ValueError(f"Index file {index_path} not found")

        with open(index_path) as f:
            all_files = [line.strip() for line in f if line.strip()]

        if self._date_range is not None:
            start_str, end_str = self._date_range
            start_date = (
                datetime.strptime(start_str, DATE_TIME_FORMAT)
                if start_str.lower() != "none"
                else None
            )
            end_date = (
                datetime.strptime(end_str, DATE_TIME_FORMAT) if end_str.lower() != "none" else None
            )
            if start_date is not None or end_date is not None:
                filtered: list[str] = []
                for file in all_files:
                    try:

View on GitHub (pinned to 24c60942c5)