{"record":{"id":"285f360bf7e8f375","repo":"microsoft/garnet","slug":"slowdown-unable-to-add-trigger-to-epoch","errorCode":null,"errorMessage":"Slowdown: Unable to add trigger to epoch\n","messagePattern":"Slowdown: Unable to add trigger to epoch\n","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"libs/storage/Tsavorite/cc/src/device/light_epoch.h","lineNumber":276,"sourceCode":"    uint32_t i = 0, j = 0;\n    while(true) {\n      uint64_t trigger_epoch = drain_list_[i].epoch.load();\n      if(trigger_epoch == EpochAction::kFree) {\n        if(drain_list_[i].TryPush(prior_epoch, callback, context)) {\n          ++drain_count_;\n          break;\n        }\n      } else if(trigger_epoch <= safe_to_reclaim_epoch.load()) {\n        if(drain_list_[i].TrySwap(trigger_epoch, prior_epoch, callback, context)) {\n          break;\n        }\n      }\n      if(++i == kDrainListSize) {\n        i = 0;\n        if(++j == 500) {\n          j = 0;\n          std::this_thread::sleep_for(std::chrono::seconds(1));\n          fprintf(stderr, \"Slowdown: Unable to add trigger to epoch\\n\");\n        }\n      }\n    }\n    return prior_epoch + 1;\n  }\n\n  /// Compute latest epoch that is safe to reclaim, by scanning the epoch table\n  uint64_t ComputeNewSafeToReclaimEpoch(uint64_t current_epoch_) {\n    uint64_t oldest_ongoing_call = current_epoch_;\n    for(uint32_t index = 0; index < kTableSize; ++index) {\n      uint64_t entry_epoch = table_[index].local_current_epoch;\n      if(entry_epoch != kUnprotected && entry_epoch < oldest_ongoing_call) {\n        oldest_ongoing_call = entry_epoch;\n      }\n    }\n    safe_to_reclaim_epoch = oldest_ongoing_call - 1;\n    return safe_to_reclaim_epoch;\n  }","sourceCodeStart":258,"sourceCodeEnd":294,"githubUrl":"https://github.com/microsoft/garnet/blob/951b0fc6838721f89d102c2bbe1b914e8d39d700/libs/storage/Tsavorite/cc/src/device/light_epoch.h#L258-L294","documentation":"Not an exception but a stderr 'Slowdown' diagnostic from LightEpoch::BumpCurrentEpoch(callback, context). After bumping the global epoch it must enqueue a reclaim-trigger into the fixed 256-entry drain_list (kDrainListSize). If every full sweep (256 slots) fails to find a free or already-reclaimable slot, a counter (j) increments; every 500 sweeps (≈128000 iterations) it sleeps 1s and prints this line, then loops forever until a slot opens. It indicates epoch-based reclamation is stalled because safe_to_reclaim_epoch is not advancing.","triggerScenarios":"A thread is stuck inside a protected critical section (Protect/ProtectAndDrain/ReentrantProtect called without the matching Unprotect/ReentrantUnprotect), so ComputeNewSafeToReclaimEpoch cannot advance safe_to_reclaim_epoch, so drain_list entries never become reclaimable and the list saturates. Also triggered by a stale entry in the thread table: a thread that exited (or was killed) without resetting its local_current_epoch to kUnprotected leaves an artificially old epoch blocking reclamation.","commonSituations":"Long-running operations holding an epoch through blocking I/O or locks; an exception path that skipped Unprotect(); thread teardown that forgot to unregister; exceeding the intended concurrency so many threads hold epochs simultaneously; a checkpoint/grow/compaction storm that enqueues reclaim triggers faster than threads drain them.","solutions":["Audit every Protect/ProtectAndDrain/ReentrantProtect for a matching Unprotect/ReentrantUnprotect on ALL exit paths (exceptions, early returns, cancellation) — prefer an RAII guard that calls Unprotect in its destructor.","Ensure threads unregister from Thread::id() / reset their epoch table entry before exiting, so no stale local_current_epoch blocks ComputeNewSafeToReclaimEpoch.","Keep protected critical sections short and non-blocking — never sleep, do disk I/O, or take coarse locks while holding an epoch.","If legitimately enqueuing more than ~256 concurrent reclaim triggers, raise kDrainListSize or batch triggers; if the stall is from one stuck thread, attach a debugger to find the thread whose local_current_epoch is oldest."],"exampleFix":"// before:\n// epoch.ProtectAndDrain();\n// DoWork();            // if this throws, Unprotect never runs -> stall\n// epoch.Unprotect();\n// after (RAII so every exit path releases the epoch):\nstruct EpochGuard {\n  LightEpoch& e; EpochGuard(LightEpoch& e_) : e(e_) { e.ProtectAndDrain(); }\n  ~EpochGuard() { e.Unprotect(); }\n};\n{ EpochGuard g{epoch}; DoWork(); }","handlingStrategy":"validation","validationCode":"// Debug-only invariant: every Protect must have a matching Unprotect; assert the\n// thread is not already protected and that no protected section blocks.\n#ifdef DEBUG\nstruct EpochUse {\n  LightEpoch& e; const char* tag;\n  EpochUse(LightEpoch& e_, const char* t) : e(e_), tag(t) {\n    assert(!e.IsProtected() && \"re-protect without unprotect\"); e.ProtectAndDrain();\n  }\n  ~EpochUse() { e.Unprotect(); }\n};\n#endif","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pair Protect/ProtectAndDrain/ReentrantProtect with the matching Unprotect on every exit path; use an RAII guard so exceptions can't skip it.","Keep protected sections short and never blocking (no I/O, sleeps, or coarse locks inside).","Unregister/reset the thread's epoch table entry before thread exit so safe_to_reclaim_epoch can advance.","Watch for the 'Slowdown: Unable to add trigger to epoch' line as the early signal of a leaked Protect; snapshot table_ entries to find the oldest local_current_epoch."],"tags":["tsavorite","cpp","epoch","reclamation","stall","deadlock","concurrency"],"backgroundTag":null,"analyzedSha":"951b0fc6838721f89d102c2bbe1b914e8d39d700","analyzedAt":"2026-08-13T19:01:32.939Z","schemaVersion":2},"datasetVersion":"2026-08-13T19:17:28.613Z"}