What causes "no cached cls parquet under {split_dir}" in xai-org/x-algorithm?

List split_dir and confirm *.parquet files exist at that exact path

xai-org/x-algorithm · error · FileNotFoundError

no cached cls parquet under {split_dir}

Error message

no cached cls parquet under {split_dir}

What it means

load_cached expects precomputed CLS-embedding parquet shards under split_dir; if glob('*.parquet') finds nothing it raises FileNotFoundError so callers do not silently train on zero rows.

Source

Thrown at bdsm/training/train_head.py:38

import pyarrow as pa
import pyarrow.parquet as pq

import heads
import labels
import loss as task_loss
from metrics import average_precision
from task_heads import SEED, head_logits, init_head_params

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s", datefmt="%H:%M:%S")
log = logging.getLogger("train_head")


def load_cached(split_dir: str) -> dict:
    xs, ys, uids, sources = [], [], [], []
    n_nonfinite = 0
    files = sorted(glob.glob(os.path.join(split_dir, "*.parquet")))
    if not files:
        raise FileNotFoundError(f"no cached cls parquet under {split_dir}")
    for fpath in files:
        t = pq.read_table(fpath)
        strength = np.asarray(t.column("label_strength").to_pylist())
        cls_col = t.column("cls").combine_chunks()
        cls_all = (
            cls_col.flatten()
            .to_numpy(zero_copy_only=False)
            .reshape(len(cls_col), -1)
            .astype(np.float32)
        )
        finite = np.isfinite(cls_all).all(axis=1)
        n_nonfinite += int((~finite).sum())
        keep = (strength == "strong") & finite
        cls = cls_all[keep]
        lbls = [lb for lb, k in zip(t.column("labels").to_pylist(), keep, strict=True) if k]
        xs.append(cls)
        ys.append(np.stack([labels.labels_to_vectors(lb)[0] for lb in lbls]))
        uids.append(np.asarray(t.column("user_id").to_pylist(), dtype=np.int64)[keep])

View on GitHub (pinned to 24c60942c5)

Solutions

List split_dir and confirm *.parquet files exist at that exact path
Run the upstream feature-extraction job that writes the cls parquet shards
Fix the split_dir argument (commonly a missing train/val subdirectory level)
Wait for/retry if shards are still being uploaded by a producer

Example fix

# before
load_cached('/data/cls')
# after
load_cached('/data/cls/train')  # dir containing part-*.parquet

Defensive patterns

Strategy: validation

Validate before calling

import glob
assert glob.glob(os.path.join(split_dir, '*.parquet')), f"no parquet in {split_dir}"

Type guard

def split_cached(split_dir: str) -> bool:
    return bool(glob.glob(os.path.join(split_dir, '*.parquet')))

Try / catch

try:
    data = load_cached(split_dir)
except FileNotFoundError:
    run_extraction(split_dir)
    data = load_cached(split_dir)

Prevention

Make extraction a prerequisite step enforced by the training DAG
Check for shard files plus a completion marker before training
Log resolved absolute split paths at startup

When it happens

Trigger: Calling load_cached('/data/train') when the extraction step has not run or wrote to a sibling dir; split_dir misspelled; parquet files named with a different extension or nested one level deeper; empty split directory in a fresh checkout.

Common situations: Running train_head before the cache-building job; wrong split path ('train' vs 'train_cls'); artifacts on a volume not mounted; shards still uploading.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/68657c71a552645d. Report an issue: GitHub.