huggingface/open-r1 · error

CF_TESTS_FOLDER environment variable not set! Please downloa

Error message

CF_TESTS_FOLDER environment variable not set! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information.

What it means

get_generated_contest_tests requires the CF_TESTS_FOLDER environment variable to locate locally downloaded Codeforces generated test parquet files. If the variable is unset, it raises ValueError with a link to the open-r1/codeforces dataset explaining how to download the tests. Without the folder, generated (hidden) test cases cannot be added to official tests for scoring.

Source

Thrown at src/open_r1/utils/competitive_programming/cf_scoring.py:67

            language="cf_python3" if submission_language == "python" else "c++17",
        )
    except Exception as e:
        print(f"Error scoring submission: {e}")
        return False

    return result


@alru_cache(maxsize=32)  # TODO make this configurable
async def get_generated_contest_tests(contest_id: str) -> list[dict]:
    import pandas as pd

    import aiofiles
    import aiofiles.os

    tests_folder = os.environ.get("CF_TESTS_FOLDER", None)
    if not tests_folder:
        raise ValueError(
            "CF_TESTS_FOLDER environment variable not set! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information."
        )
    if not await aiofiles.os.path.exists(tests_folder):
        raise ValueError(
            f"CF_TESTS_FOLDER path '{tests_folder}' does not exist! Please download the codeforces generated tests and set CF_TESTS_FOLDER to the folder path. See https://huggingface.co/datasets/open-r1/codeforces for more information."
        )
    parquet_path = os.path.join(tests_folder, f"test_cases_{int(contest_id):04d}.parquet")
    if not await aiofiles.os.path.exists(parquet_path):
        return {}

    # Read parquet file asynchronously
    async with aiofiles.open(parquet_path, "rb") as f:
        content = await f.read()
        df = pd.read_parquet(BytesIO(content))

    # Group by problem_id and convert to dictionary of lists
    grouped_tests = df.groupby("problem_id").apply(lambda x: x[["input", "output"]].to_dict("records")).to_dict()

View on GitHub (pinned to 1416fa0cf2)

Solutions

1. Download the codeforces generated tests from https://huggingface.co/datasets/open-r1/codeforces
2. Set CF_TESTS_FOLDER to the absolute path of the folder containing the test_cases_*.parquet files before launching
3. Persist it via .env, sbatch --export, or CI secret so worker nodes see it
4. Verify: CF_TESTS_FOLDER=/path python -c "import os; print(os.environ['CF_TESTS_FOLDER'])"

Example fix

// before
score = await score_submission(problem, code)  # ValueError: CF_TESTS_FOLDER not set
// after
# shell: export CF_TESTS_FOLDER=/data/codeforces_tests
score = await score_submission(problem, code)

import os

def require_cf_tests_folder():
    path = os.environ.get("CF_TESTS_FOLDER")
    if not path:
        raise SystemExit(
            "Set CF_TESTS_FOLDER to the folder with codeforces test parquet files "
            "(see https://huggingface.co/datasets/open-r1/codeforces)"
        )
    if not os.path.isdir(path):
        raise SystemExit(f"CF_TESTS_FOLDER {path!r} does not exist")

def cf_tests_env_ready() -> bool:
    import os
    return bool(os.environ.get("CF_TESTS_FOLDER"))

try:
    tests = await get_generated_tests(problem_id)
except ValueError as e:
    if "CF_TESTS_FOLDER" in str(e):
        logger.error("%s — export CF_TESTS_FOLDER before running", e)
        raise SystemExit(1) from e
    raise

When it happens

Trigger: Calling get_generated_tests / score_submission when CF_TESTS_FOLDER is not present in os.environ at scoring time.

Common situations: Running evaluation on a cluster/CI node where .env or export of CF_TESTS_FOLDER wasn't propagated; downloading the dataset to a laptop but running scoring elsewhere; new contributor following README partially; env var named incorrectly (CF_TEST_FOLDER).

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• Either `dataset_name` or `dataset_mixture` must be provided
• dataset_mixture must be a dictionary with a 'datasets' key.
• 'datasets' must be a list of dataset configurations
• Column names must be consistent across all dataset configura
• MorphCloud API key not found. Please set the MORPH_API_KEY e

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/bd37e0de958c1585. Report an issue: GitHub.