xai-org/x-algorithm · error · ValueError

Cannot load metadata from {metadata_path}

Error message

Cannot load metadata from {metadata_path}

What it means

When a time range is requested, __init__ calls _load_valid_batches_metadata(metadata_path); if it returns None (file missing, unreadable, or malformed JSON), it raises 'Cannot load metadata from {path}'. The metadata file is required to map timestamps onto valid batch ids.

Source

Thrown at phoenix/xrex/data/parquet_recsys.py:308

        self._batch_size = batch_size
        self._num_shards = num_shards
        self._shard_index = shard_index
        self._interleave_k = interleave_k
        self._date_range = date_range
        self._continuous = continuous
        self._poll_interval_s = poll_interval_s
        self._num_kafka_partitions = num_kafka_partitions

        self._end_batch_id: int | None = None
        start_batch_id = 0

        self._remaining_skips: int = 0

        if has_time_range:
            assert metadata_path is not None
            meta = _load_valid_batches_metadata(metadata_path)
            if meta is None:
                raise ValueError(f"Cannot load metadata from {metadata_path}")
            start_batch_id, end_batch_id = _resolve_time_range(
                self._topic_dir,
                meta["min_valid_batch"],
                meta["max_valid_batch"],
                min_timestamp_ms,
                max_timestamp_ms,
            )
            if max_timestamp_ms is not None:
                self._end_batch_id = end_batch_id

        if resume_position is not None:
            resume_bid = resume_position["last_batch_id"]
            resume_in_range = resume_bid >= start_batch_id and (
                self._end_batch_id is None or resume_bid < self._end_batch_id
            )
            if resume_in_range:
                self._next_batch_id = resume_bid
                saved_reads = resume_position["rows_read_in_batch"]

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify the file exists and is valid JSON: python -c "import json;json.load(open(p))".
2. Regenerate .valid_batches.json from the topic directory with the metadata-producing job.
3. Check metadata_path is absolute or correctly relative to the working directory.

Example fix

# before
ds = ParquetRecsysDataset(..., metadata_path='/data/topic.valid_batches.json', min_timestamp_ms=t0)  # ValueError

# after (regenerate then retry)
$ xrex make-valid-batches --topic-dir /data/topic
 ds = ParquetRecsysDataset(..., metadata_path='/data/topic.valid_batches.json', min_timestamp_ms=t0)

import json, os
if has_time_range:
    assert metadata_path and os.path.isfile(metadata_path), f'missing {metadata_path}'
    json.load(open(metadata_path))  # parse check

try:
    ds = ParquetRecsysDataset(...)
except ValueError as e:
    if 'Cannot load metadata' in str(e):
        regenerate_metadata(topic_dir); retry()
    else:
        raise

When it happens

Trigger: metadata_path points to a nonexistent or empty .valid_batches.json; JSON corrupted by a concurrent writer; wrong topic_dir making the relative metadata path unresolvable.

Common situations: Metadata file not yet generated by the pipeline; partial copy of a data directory; permission issues on the metadata file.

Related errors

• Did not find any files matching {file_pattern}
• no cached cls parquet under {split_dir}
• Keytab file not found: {self.config.keytab_path}
• min_timestamp_ms/max_timestamp_ms require metadata mode (.va
• Index file {index_path} not found

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/6b3b5e5f4b168788. Report an issue: GitHub.