huggingface/open-r1 · error

CF_TESTS_FOLDER path '{tests_folder}' does not exist! Please

Error message

CF_TESTS_FOLDER path '{tests_folder}' does not exist! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information.

What it means

This is the follow-up check to error 16: CF_TESTS_FOLDER is set, but the path it points to does not exist on the filesystem (checked asynchronously with aiofiles.os.path.exists). The same remediation guidance is embedded in the message, including the dataset link.

Source

Thrown at src/open_r1/utils/competitive_programming/cf_scoring.py:71

        return False

    return result


@alru_cache(maxsize=32)  # TODO make this configurable
async def get_generated_contest_tests(contest_id: str) -> list[dict]:
    import pandas as pd

    import aiofiles
    import aiofiles.os

    tests_folder = os.environ.get("CF_TESTS_FOLDER", None)
    if not tests_folder:
        raise ValueError(
            "CF_TESTS_FOLDER environment variable not set! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information."
        )
    if not await aiofiles.os.path.exists(tests_folder):
        raise ValueError(
            f"CF_TESTS_FOLDER path '{tests_folder}' does not exist! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information."
        )
    parquet_path = os.path.join(tests_folder, f"test_cases_{int(contest_id):04d}.parquet")
    if not await aiofiles.os.path.exists(parquet_path):
        return {}

    # Read parquet file asynchronously
    async with aiofiles.open(parquet_path, "rb") as f:
        content = await f.read()
        df = pd.read_parquet(BytesIO(content))

    # Group by problem_id and convert to dictionary of lists
    grouped_tests = df.groupby("problem_id").apply(lambda x: x[["input", "output"]].to_dict("records")).to_dict()

    return grouped_tests


async def get_generated_tests(problem_id: str) -> list[dict]:

View on GitHub (pinned to 1416fa0cf2)

Solutions

1. Verify the path exists: ls "$CF_TESTS_FOLDER"
2. Use an absolute path that resolves identically on all nodes (shared/network filesystem for clusters)
3. Re-download the test parquet files from https://huggingface.co/datasets/open-r1/codeforces
4. Check the value for typos, whitespace, or wrong username/volume paths; confirm cwd assumptions for relative paths

Example fix

// before
# .env: CF_TESTS_FOLDER=./tests  (dir absent on compute node)
score = await score_submission(problem, code)  # ValueError: path does not exist
// after
# .env: CF_TESTS_FOLDER=/mnt/shared/codeforces_tests
score = await score_submission(problem, code)

import os
from pathlib import Path

def require_cf_tests_path():
    raw = os.environ.get("CF_TESTS_FOLDER", "")
    path = Path(raw.strip()).expanduser().resolve()
    if not path.is_dir():
        raise SystemExit(
            f"CF_TESTS_FOLDER resolves to {path}, which does not exist. "
            "Download tests from https://huggingface.co/datasets/open-r1/codeforces"
        )
    if not any(path.glob("test_cases_*.parquet")):
        raise SystemExit(f"No test_cases_*.parquet files found in {path}")

def cf_tests_folder_exists() -> bool:
    import os
    p = os.environ.get("CF_TESTS_FOLDER")
    return bool(p) and os.path.isdir(p)

try:
    tests = await get_generated_tests(problem_id)
except ValueError as e:
    if "does not exist" in str(e):
        logger.error("%s — check path, mounts, and that tests were downloaded on this node", e)
        raise SystemExit(1) from e
    raise

When it happens

Trigger: get_generated_contest_tests runs with CF_TESTS_FOLDER set to a path that fails the exists() check — deleted/moved folder, relative path resolved against a different working directory, or the tests downloaded on another machine/volume not mounted on the node.

Common situations: Relative path in .env that doesn't resolve from the job's cwd; cluster node without the shared filesystem mounted; folder deleted to free disk space; typo in the path or trailing whitespace in the env value; missing download step on a new machine.

Related errors

• Either `dataset_name` or `dataset_mixture` must be provided
• dataset_mixture must be a dictionary with a 'datasets' key.
• 'datasets' must be a list of dataset configurations
• Column names must be consistent across all dataset configura
• Callback {callback_name} not found in CALLBACKS.

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/d5f8da3afc5b144e. Report an issue: GitHub.