{"record":{"id":"6b3b5e5f4b168788","repo":"xai-org/x-algorithm","slug":"cannot-load-metadata-from-metadata-path","errorCode":null,"errorMessage":"Cannot load metadata from {metadata_path}","messagePattern":"Cannot load metadata from (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/data/parquet_recsys.py","lineNumber":308,"sourceCode":"        self._batch_size = batch_size\n        self._num_shards = num_shards\n        self._shard_index = shard_index\n        self._interleave_k = interleave_k\n        self._date_range = date_range\n        self._continuous = continuous\n        self._poll_interval_s = poll_interval_s\n        self._num_kafka_partitions = num_kafka_partitions\n\n        self._end_batch_id: int | None = None\n        start_batch_id = 0\n\n        self._remaining_skips: int = 0\n\n        if has_time_range:\n            assert metadata_path is not None\n            meta = _load_valid_batches_metadata(metadata_path)\n            if meta is None:\n                raise ValueError(f\"Cannot load metadata from {metadata_path}\")\n            start_batch_id, end_batch_id = _resolve_time_range(\n                self._topic_dir,\n                meta[\"min_valid_batch\"],\n                meta[\"max_valid_batch\"],\n                min_timestamp_ms,\n                max_timestamp_ms,\n            )\n            if max_timestamp_ms is not None:\n                self._end_batch_id = end_batch_id\n\n        if resume_position is not None:\n            resume_bid = resume_position[\"last_batch_id\"]\n            resume_in_range = resume_bid >= start_batch_id and (\n                self._end_batch_id is None or resume_bid < self._end_batch_id\n            )\n            if resume_in_range:\n                self._next_batch_id = resume_bid\n                saved_reads = resume_position[\"rows_read_in_batch\"]","sourceCodeStart":290,"sourceCodeEnd":326,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/data/parquet_recsys.py#L290-L326","documentation":"When a time range is requested, __init__ calls _load_valid_batches_metadata(metadata_path); if it returns None (file missing, unreadable, or malformed JSON), it raises 'Cannot load metadata from {path}'. The metadata file is required to map timestamps onto valid batch ids.","triggerScenarios":"metadata_path points to a nonexistent or empty .valid_batches.json; JSON corrupted by a concurrent writer; wrong topic_dir making the relative metadata path unresolvable.","commonSituations":"Metadata file not yet generated by the pipeline; partial copy of a data directory; permission issues on the metadata file.","solutions":["Verify the file exists and is valid JSON: python -c \"import json;json.load(open(p))\".","Regenerate .valid_batches.json from the topic directory with the metadata-producing job.","Check metadata_path is absolute or correctly relative to the working directory."],"exampleFix":"# before\nds = ParquetRecsysDataset(..., metadata_path='/data/topic.valid_batches.json', min_timestamp_ms=t0)  # ValueError\n\n# after (regenerate then retry)\n$ xrex make-valid-batches --topic-dir /data/topic\n ds = ParquetRecsysDataset(..., metadata_path='/data/topic.valid_batches.json', min_timestamp_ms=t0)","handlingStrategy":"validation","validationCode":"import json, os\nif has_time_range:\n    assert metadata_path and os.path.isfile(metadata_path), f'missing {metadata_path}'\n    json.load(open(metadata_path))  # parse check","typeGuard":null,"tryCatchPattern":"try:\n    ds = ParquetRecsysDataset(...)\nexcept ValueError as e:\n    if 'Cannot load metadata' in str(e):\n        regenerate_metadata(topic_dir); retry()\n    else:\n        raise","preventionTips":["Make metadata generation a prerequisite step in the training DAG.","Health-check the metadata file (exists, parses, has expected keys) before jobs start."],"tags":["metadata","file-not-found","json","time-range"],"backgroundTag":"metadata-file-unreadable","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}