{"record":{"id":"a2e6cad19c8b59f6","repo":"hsliuping/TradingAgents-CN","slug":"reddit-fetching-error-max-limit-is-less-than-the","errorCode":null,"errorMessage":"REDDIT FETCHING ERROR: max limit is less than the number of files in the category. Will not be able to fetch any posts","messagePattern":"REDDIT FETCHING ERROR: max limit is less than the number of files in the category\\. Will not be able to fetch any posts","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"tradingagents/dataflows/news/reddit.py","lineNumber":69,"sourceCode":"\ndef fetch_top_from_category(\n    category: Annotated[\n        str, \"Category to fetch top post from. Collection of subreddits.\"\n    ],\n    date: Annotated[str, \"Date to fetch top posts from.\"],\n    max_limit: Annotated[int, \"Maximum number of posts to fetch.\"],\n    query: Annotated[str, \"Optional query to search for in the subreddit.\"] = None,\n    data_path: Annotated[\n        str,\n        \"Path to the data folder. Default is 'reddit_data'.\",\n    ] = \"reddit_data\",\n):\n    base_path = data_path\n\n    all_content = []\n\n    if max_limit < len(os.listdir(os.path.join(base_path, category))):\n        raise ValueError(\n            \"REDDIT FETCHING ERROR: max limit is less than the number of files in the category. Will not be able to fetch any posts\"\n        )\n\n    limit_per_subreddit = max_limit // len(\n        os.listdir(os.path.join(base_path, category))\n    )\n\n    for data_file in os.listdir(os.path.join(base_path, category)):\n        # check if data_file is a .jsonl file\n        if not data_file.endswith(\".jsonl\"):\n            continue\n\n        all_content_curr_subreddit = []\n\n        with open(os.path.join(base_path, category, data_file), \"rb\") as f:\n            for i, line in enumerate(f):\n                # skip empty lines\n                if not line.strip():","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/hsliuping/TradingAgents-CN/blob/74783e8817d6cf6de29867880631cc555153f36b/tradingagents/dataflows/news/reddit.py#L51-L87","documentation":"Raised by fetch_top_from_category when max_limit is smaller than the number of files in the local reddit category directory. The function divides max_limit evenly across subreddit files, so fewer posts than files would yield zero posts per subreddit. NOTE: the comparison is inverted relative to the message wording — it fires when max_limit < file count.","triggerScenarios":"Calling get_reddit_global_news(..., max_limit=5) when the category dir (e.g. data_path/reddit/global) contains more than 5 files; small max_limit values with many cached subreddit JSON files.","commonSituations":"Lowering the post limit for faster tests, adding more subreddit files to the data directory, or a changed default data_path pointing at a larger directory.","solutions":["Increase max_limit to at least the number of files in the category directory (e.g. 50)","Count files first: len(os.listdir(os.path.join(data_path, category))) and pass max_limit >= that","Remove unused subreddit files from the directory if you truly need few posts"],"exampleFix":"# before\nposts = get_reddit_global_news('2025-01-01', max_limit=5)\n# after\nposts = get_reddit_global_news('2025-01-01', max_limit=100)","handlingStrategy":"validation","validationCode":"import os\nn_files = len(os.listdir(os.path.join(data_path, category)))\nmax_limit = max(max_limit, n_files)\nposts = fetch_top_from_category(data_path, category, max_limit)","typeGuard":null,"tryCatchPattern":"try:\n    posts = get_reddit_global_news(curr_date, max_limit)\nexcept ValueError as e:\n    if 'max limit' in str(e):\n        posts = get_reddit_global_news(curr_date, 100)\n    else:\n        raise","preventionTips":["Default max_limit generously (>= number of subreddit files)","Compute limits from directory contents at runtime","Pin the reddit dataset directory contents in deployments"],"tags":["python","reddit","offline-data","validation"],"backgroundTag":"invalid-argument-combination","analyzedSha":"74783e8817d6cf6de29867880631cc555153f36b","analyzedAt":"2026-08-28T11:39:07.729Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}