microsoft/FASTER · error · FasterException

Duplicate tag found in index

Error message

Duplicate tag found in index

What it means

During index traversal (e.g., used by index verification/counting logic in FasterKV), the code iterates hash bucket entries and collects tags; if the same tag is found on two distinct valid entries (and the entry is not tentative), the index is inconsistent, so it throws. Tentative entries are allowed because they are transient during pending operations.

Solutions

1. Rebuild the index by recovering from a clean hybrid-log checkpoint (index is fully reconstructible from the log)
2. Recover from an earlier checkpoint token known to be consistent
3. Stop concurrent checkpoint/resize operations and re-run the scan under proper epoch protection to rule out transient issues
4. If reproducible, file with memory corruption checks (e.g., enable managed heap checks / re-run with smaller workload)

Example fix

// before: scanning index while operations are running
fasterKV.GrowIndex();
var count = fasterKV.IndexSize; // scanning paths may hit duplicate tags mid-mutation
// after: quiesce sessions/checkpoints before verification
await fasterKV.CompleteCheckpointAsync();
// then scan/verify index with no concurrent writers

// Ensure no checkpoint/resize is in flight before scanning the index
if (fasterKV.CheckpointState.IsActive()) // or equivalent state check
    await fasterKV.CompleteCheckpointAsync(); // quiesce first

try { ScanIndexForVerification(); }
catch (FasterException ex) when (ex.Message == "Duplicate tag found in index")
{
    // rebuild the index from the hybrid log via recovery from a clean checkpoint
    fasterKV.Recover(lastCleanCheckpointToken);
}

When it happens

Trigger: Scanning the hash index when it contains a corrupted or duplicated tag entry: index file corrupted on disk, restore of an index checkpoint with duplicate tags, memory corruption, or scanning the index concurrently without epoch protection while entries change.

Common situations: Diagnosing suspected index corruption after crashes; verifying index checkpoint recovery; running index scans while checkpoints/resizes are in flight.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Unable to set first valid segment to
• Unable to set last valid segment to
• Can spin-wait for commit (checkpoint completion) only if…
• Make sure all async operations issued on this session are…
• Metadata of size does not fit in delta log space of size

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/428085a1b1c4fb9b. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/core/Index/FASTER/FASTER.cs:887

            long total_record_count = 0;
            long beginAddress = hlog.BeginAddress;
            Dictionary<int, long> histogram = new();

            for (long bucket = 0; bucket < table_size_; ++bucket)
            {
                List<int> tags = new();
                int cnt = 0;
                HashBucket b = *(ptable_ + bucket);
                while (true)
                {
                    for (int bucket_entry = 0; bucket_entry < Constants.kOverflowBucketIndex; ++bucket_entry)
                    {
                        var x = default(HashBucketEntry);
                        x.word = b.bucket_entries[bucket_entry];
                        if (((!x.ReadCache) && (x.Address >= beginAddress)) || (x.ReadCache && (x.AbsoluteAddress >= readcache.HeadAddress)))
                        {
                            if (tags.Contains(x.Tag) && !x.Tentative)
                                throw new FasterException("Duplicate tag found in index");
                            tags.Add(x.Tag);
                            ++cnt;
                            ++total_record_count;
                        }
                    }
                    if ((b.bucket_entries[Constants.kOverflowBucketIndex] & Constants.kAddressMask) == 0) break;
                    b = *(HashBucket*)overflowBucketsAllocator.GetPhysicalAddress(b.bucket_entries[Constants.kOverflowBucketIndex] & Constants.kAddressMask);
                }

                if (!histogram.ContainsKey(cnt)) histogram[cnt] = 0;
                histogram[cnt]++;
            }

            var distribution =
                $"Number of hash buckets: {table_size_}\n" +
                $"Number of overflow buckets: {OverflowBucketCount}\n" +
                $"Size of each bucket: {Constants.kEntriesPerBucket * sizeof(HashBucketEntry)} bytes\n" +
                $"Total distinct hash-table entry count: {{{total_record_count}}}\n" +

View on GitHub (pinned to 321d872eab)