{"record":{"id":"428085a1b1c4fb9b","repo":"microsoft/FASTER","slug":"duplicate-tag-found-in-index","errorCode":null,"errorMessage":"Duplicate tag found in index","messagePattern":"Duplicate tag found in index","errorType":"exception","errorClass":"FasterException","httpStatus":null,"severity":"error","filePath":"cs/src/core/Index/FASTER/FASTER.cs","lineNumber":887,"sourceCode":"            long total_record_count = 0;\n            long beginAddress = hlog.BeginAddress;\n            Dictionary<int, long> histogram = new();\n\n            for (long bucket = 0; bucket < table_size_; ++bucket)\n            {\n                List<int> tags = new();\n                int cnt = 0;\n                HashBucket b = *(ptable_ + bucket);\n                while (true)\n                {\n                    for (int bucket_entry = 0; bucket_entry < Constants.kOverflowBucketIndex; ++bucket_entry)\n                    {\n                        var x = default(HashBucketEntry);\n                        x.word = b.bucket_entries[bucket_entry];\n                        if (((!x.ReadCache) && (x.Address >= beginAddress)) || (x.ReadCache && (x.AbsoluteAddress >= readcache.HeadAddress)))\n                        {\n                            if (tags.Contains(x.Tag) && !x.Tentative)\n                                throw new FasterException(\"Duplicate tag found in index\");\n                            tags.Add(x.Tag);\n                            ++cnt;\n                            ++total_record_count;\n                        }\n                    }\n                    if ((b.bucket_entries[Constants.kOverflowBucketIndex] & Constants.kAddressMask) == 0) break;\n                    b = *(HashBucket*)overflowBucketsAllocator.GetPhysicalAddress(b.bucket_entries[Constants.kOverflowBucketIndex] & Constants.kAddressMask);\n                }\n\n                if (!histogram.ContainsKey(cnt)) histogram[cnt] = 0;\n                histogram[cnt]++;\n            }\n\n            var distribution =\n                $\"Number of hash buckets: {table_size_}\\n\" +\n                $\"Number of overflow buckets: {OverflowBucketCount}\\n\" +\n                $\"Size of each bucket: {Constants.kEntriesPerBucket * sizeof(HashBucketEntry)} bytes\\n\" +\n                $\"Total distinct hash-table entry count: {{{total_record_count}}}\\n\" +","sourceCodeStart":869,"sourceCodeEnd":905,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cs/src/core/Index/FASTER/FASTER.cs#L869-L905","documentation":"During index traversal (e.g., used by index verification/counting logic in FasterKV), the code iterates hash bucket entries and collects tags; if the same tag is found on two distinct valid entries (and the entry is not tentative), the index is inconsistent, so it throws. Tentative entries are allowed because they are transient during pending operations.","triggerScenarios":"Scanning the hash index when it contains a corrupted or duplicated tag entry: index file corrupted on disk, restore of an index checkpoint with duplicate tags, memory corruption, or scanning the index concurrently without epoch protection while entries change.","commonSituations":"Diagnosing suspected index corruption after crashes; verifying index checkpoint recovery; running index scans while checkpoints/resizes are in flight.","solutions":["Rebuild the index by recovering from a clean hybrid-log checkpoint (index is fully reconstructible from the log)","Recover from an earlier checkpoint token known to be consistent","Stop concurrent checkpoint/resize operations and re-run the scan under proper epoch protection to rule out transient issues","If reproducible, file with memory corruption checks (e.g., enable managed heap checks / re-run with smaller workload)"],"exampleFix":"// before: scanning index while operations are running\nfasterKV.GrowIndex();\nvar count = fasterKV.IndexSize; // scanning paths may hit duplicate tags mid-mutation\n// after: quiesce sessions/checkpoints before verification\nawait fasterKV.CompleteCheckpointAsync();\n// then scan/verify index with no concurrent writers","handlingStrategy":"try-catch","validationCode":"// Ensure no checkpoint/resize is in flight before scanning the index\nif (fasterKV.CheckpointState.IsActive()) // or equivalent state check\n    await fasterKV.CompleteCheckpointAsync(); // quiesce first","typeGuard":null,"tryCatchPattern":"try { ScanIndexForVerification(); }\ncatch (FasterException ex) when (ex.Message == \"Duplicate tag found in index\")\n{\n    // rebuild the index from the hybrid log via recovery from a clean checkpoint\n    fasterKV.Recover(lastCleanCheckpointToken);\n}","preventionTips":["Do not scan the hash index while concurrent writers, checkpoints, or resizes run","Maintain periodic full checkpoints as a clean recovery point","Investigate duplicate-tag reports as possible corruption; check storage health","Use tentative-entry-aware tooling if scanning during transient operation"],"tags":["index-corruption","consistency","checkpoint"],"backgroundTag":"internal-invariant-violation","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}