{"record":{"id":"386308a9da02d2fb","repo":"microsoft/qlib","slug":"no-enough-data-for-calculating-ic","errorCode":null,"errorMessage":"No enough data for calculating IC","messagePattern":"No enough data for calculating IC","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"qlib/contrib/meta/data_selection/utils.py","lineNumber":58,"sourceCode":"            if pred_focus.shape[0] < self.skip_size:\n                # skip some days which have very small amount of stock.\n                skip_n += 1\n                continue\n            y_focus = y[start_i:end_i]\n            if pred_focus.std() < EPS or y_focus.std() < EPS:\n                # These cases often happend at the end of test data.\n                # Usually caused by fillna(0.)\n                skip_n += 1\n                continue\n\n            ic_day = torch.dot(\n                (pred_focus - pred_focus.mean()) / np.sqrt(pred_focus.shape[0]) / pred_focus.std(),\n                (y_focus - y_focus.mean()) / np.sqrt(y_focus.shape[0]) / y_focus.std(),\n            )\n            ic_all += ic_day\n        if len(diff_point) - 1 - skip_n <= 0:\n            __import__(\"ipdb\").set_trace()\n            raise ValueError(\"No enough data for calculating IC\")\n        if skip_n > 0:\n            get_module_logger(\"ICLoss\").info(\n                f\"{skip_n} days are skipped due to zero std or small scale of valid samples.\"\n            )\n        ic_mean = ic_all / (len(diff_point) - 1 - skip_n)\n        return -ic_mean  # ic loss\n\n\ndef preds_to_weight_with_clamp(preds, clip_weight=None, clip_method=\"tanh\"):\n    \"\"\"\n    Clip the weights.\n\n    Parameters\n    ----------\n    clip_weight: float\n        The clip threshold.\n    clip_method: str\n        The clip method. Current available: \"clamp\", \"tanh\", and \"sigmoid\".","sourceCodeStart":40,"sourceCodeEnd":76,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/qlib/contrib/meta/data_selection/utils.py#L40-L76","documentation":"ICLoss.forward slices predictions/labels per day (via index boundaries in idx) and skips days with fewer than skip_size samples or zero std. If every day is skipped, the count of usable days (len(diff_point)-1-skip_n) is <= 0 and it raises ValueError. Note: the raise is preceded by __import__('ipdb').set_trace(), a leftover debug breakpoint that will first hang or crash in non-interactive environments where ipdb is not installed.","triggerScenarios":"Calling ICLoss (via MetaModelDS with criterion='ic_loss') on a test set with very few stocks per day (< skip_size=50), or where predictions/labels are constant per day (e.g. after fillna(0.0) at the tail of the data), so all days get skipped.","commonSituations":"Tiny test universes or single-stock tests; label windows running past data end so trailing rows are zero-filled; running headless (CI, docker, scheduled jobs) where the ipdb.set_trace() itself fails with ImportError or blocks forever.","solutions":["Ensure the test data passed to ICLoss has >= skip_size (default 50) valid instruments per date.","Trim the tail of the test data where labels were filled with 0.0 (zero std days are skipped).","Lower ICLoss(skip_size=...) if a smaller cross-section is expected.","Install ipdb if you must reproduce interactively; in production, patch out the set_trace() line or pin a qlib version where it is removed — otherwise the debugger hook fires before the ValueError."],"exampleFix":"// before\ncriterion = ICLoss()  # default skip_size=50; test set has 20 stocks/day -> raises\nloss = criterion(pred, y_test, test_idx)\n\n// after\ncriterion = ICLoss(skip_size=10)\nloss = criterion(pred, y_test, test_idx)","handlingStrategy":"validation","validationCode":"import collections\nday_counts = collections.Counter(idx.get_level_values(0))\nusable = [d for d, n in day_counts.items() if n >= skip_size]\nassert usable, \"no day has >= skip_size instruments; ICLoss would raise\"\nloss = criterion(pred, y, idx)","typeGuard":null,"tryCatchPattern":"try:\n    loss = criterion(pred, y_test, test_idx)\nexcept ValueError as e:\n    if \"No enough data\" in str(e):\n        continue  # MetaModelDS already catches this per-batch; mirror that pattern\n    raise","preventionTips":["Ensure >= ICLoss.skip_size (default 50) instruments per trading day in test data.","Trim zero-filled tails of labels before the loss.","Be aware the raise is preceded by __import__('ipdb').set_trace(): install ipdb or patch that line when running headless, or training may hang before the error surfaces.","Mirror MetaModelDS's own pattern: catch the ValueError, log, and skip the batch."],"tags":["qlib","meta-learning","ic-loss","data-coverage","debug-leftover"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}