CF_TESTS_FOLDER environment variable not set! Please…

Error message

CF_TESTS_FOLDER environment variable not set! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information.

What it means

get_generated_contest_tests requires the CF_TESTS_FOLDER environment variable to locate locally downloaded Codeforces generated test parquet files. If the variable is unset, it raises ValueError with a link to the open-r1/codeforces dataset explaining how to download the tests. Without the folder, generated (hidden) test cases cannot be added to official tests for scoring.

Solutions

Download the codeforces generated tests from https://huggingface.co/datasets/open-r1/codeforces
Set CF_TESTS_FOLDER to the absolute path of the folder containing the test_cases_*.parquet files before launching
Persist it via .env, sbatch --export, or CI secret so worker nodes see it
Verify: CF_TESTS_FOLDER=/path python -c "import os; print(os.environ['CF_TESTS_FOLDER'])"

Example fix

// before
score = await score_submission(problem, code)  # ValueError: CF_TESTS_FOLDER not set
// after
# shell: export CF_TESTS_FOLDER=/data/codeforces_tests
score = await score_submission(problem, code)

Defensive patterns

Strategy: validation

Validate before calling

import os

def require_cf_tests_folder():
    path = os.environ.get("CF_TESTS_FOLDER")
    if not path:
        raise SystemExit(
            "Set CF_TESTS_FOLDER to the folder with codeforces test parquet files "
            "(see https://huggingface.co/datasets/open-r1/codeforces)"
        )
    if not os.path.isdir(path):
        raise SystemExit(f"CF_TESTS_FOLDER {path!r} does not exist")

Type guard

def cf_tests_env_ready() -> bool:
    import os
    return bool(os.environ.get("CF_TESTS_FOLDER"))

Try / catch

try:
    tests = await get_generated_tests(problem_id)
except ValueError as e:
    if "CF_TESTS_FOLDER" in str(e):
        logger.error("%s — export CF_TESTS_FOLDER before running", e)
        raise SystemExit(1) from e
    raise

Prevention

Download tests once to a shared volume and export CF_TESTS_FOLDER in job templates/SLURM scripts
Add the variable to .env.example and CI secret setup docs
Preflight-check env vars at job startup for all required paths
Use absolute paths so relative-path cwd differences can't bite

When it happens

Trigger: Calling get_generated_tests / score_submission when CF_TESTS_FOLDER is not present in os.environ at scoring time.

Common situations: Running evaluation on a cluster/CI node where .env or export of CF_TESTS_FOLDER wasn't propagated; downloading the dataset to a laptop but running scoring elsewhere; new contributor following README partially; env var named incorrectly (CF_TEST_FOLDER).

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/bd37e0de958c1585. Report an issue: GitHub.

Appendix: source

Thrown at src/open_r1/utils/competitive_programming/cf_scoring.py:67

            language="cf_python3" if submission_language == "python" else "c++17",
        )
    except Exception as e:
        print(f"Error scoring submission: {e}")
        return False

    return result


@alru_cache(maxsize=32)  # TODO make this configurable
async def get_generated_contest_tests(contest_id: str) -> list[dict]:
    import pandas as pd

    import aiofiles
    import aiofiles.os

    tests_folder = os.environ.get("CF_TESTS_FOLDER", None)
    if not tests_folder:
        raise ValueError(
            "CF_TESTS_FOLDER environment variable not set! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information."
        )
    if not await aiofiles.os.path.exists(tests_folder):
        raise ValueError(
            f"CF_TESTS_FOLDER path '{tests_folder}' does not exist! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information."
        )
    parquet_path = os.path.join(tests_folder, f"test_cases_{int(contest_id):04d}.parquet")
    if not await aiofiles.os.path.exists(parquet_path):
        return {}

    # Read parquet file asynchronously
    async with aiofiles.open(parquet_path, "rb") as f:
        content = await f.read()
        df = pd.read_parquet(BytesIO(content))

    # Group by problem_id and convert to dictionary of lists
    grouped_tests = df.groupby("problem_id").apply(lambda x: x[["input", "output"]].to_dict("records")).to_dict()

View on GitHub (pinned to 1416fa0cf2)