microsoft/FASTER · warning

Expected empty entry -- found it occupied...

Error message

Expected empty entry -- found it occupied...

What it means

During hash-index checkpoint/rename (GC / relocation of entries), FASTER moves an occupied entry into a free slot. The CAS expecting the free slot to be empty failed because another thread claimed it concurrently, so the relocation of the front bucket entry is abandoned. This is an internal race log in lock-free index maintenance, typically benign for correctness but indicates heavy concurrent index mutation.

Solutions

1. Retry the operation — FASTER's internal logic continues with the next free entry; if user-visible, retry Checkpoint()
2. Reduce concurrent mutation during checkpoint (quiesce writers or use epoch barriers) if it recurs persistently
3. Upgrade FASTER — relocation race handling has been refined in later versions
4. Check epoch protection: ensure all index-mutating calls run under proper FASTER epoch guards

Example fix

null

Status s = ctx->Checkpoint(token, cb);
if (s != Status::Ok) {
  // transient race during concurrent relocation: retry once after quiescing writers
  quiesce_writers();
  s = ctx->Checkpoint(token, cb);
  resume_writers();
}

When it happens

Trigger: Concurrent Checkpoint()/GrowIndex/GC (epoch-protected entry relocation) while other threads insert/delete entries in the same hash bucket — next_free_entry's slot got occupied between finding it and the compare_exchange.

Common situations: High-concurrency workloads with many concurrent upserts/deletes during index checkpointing or garbage collection; long-running servers calling Checkpoint() under load.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Expected occupied entry -- found it empty...
• Unexpected sealed buffer found
• Invalid release by non-owner thread
• LockableUnsafeContext requires
• LockableContext requires

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/492fbd719f86d950. Report an issue: GitHub.

Appendix: source

Thrown at cc/src/index/mem_index.h:839

        AtomicHashBucketEntry* front_atomic_entry = &(front_bucket->entries[front_entry_idx]);
        hash_bucket_entry_t front_bucket_entry{ front_atomic_entry->load() };
        HashBucketEntry front_bucket_expected_entry{ front_bucket_entry };

        if (front_bucket_entry.unused()) {
          if (next_free_entry == nullptr) {
            next_free_entry = front_atomic_entry;
            back_bucket = front_bucket;
            back_entry_idx = front_entry_idx;
          }
          continue;
        }
        // Entry is occupied!

        if (next_free_entry) {
          // Move occupied entry to closest (forward-wise) free entry
          HashBucketEntry empty_entry{ HashBucketEntry::kInvalidEntry };
          if (!next_free_entry->compare_exchange_strong(empty_entry, front_bucket_expected_entry)) {
            log_error("Expected empty entry -- found it occupied...");
          } else {
            if (!front_atomic_entry->compare_exchange_strong(front_bucket_expected_entry, empty_entry)) {
              log_error("Expected occupied entry -- found it empty...");
            }
          }
          next_free_entry = nullptr;

          // Try to find next available free entry
          AtomicHashBucketEntry* back_atomic_entry;
          do {
            for (; back_entry_idx < hash_bucket_t::kNumEntries; ++back_entry_idx) {
              back_atomic_entry = &(back_bucket->entries[back_entry_idx]);
              hash_bucket_entry_t back_bucket_entry{ back_atomic_entry->load() };
              if (back_bucket_entry.unused()) {
                // Found new available entry!
                next_free_entry = back_atomic_entry;
                break;
              }

View on GitHub (pinned to 321d872eab)