{"record":{"id":"f49cb5eb26c1478c","repo":"microsoft/FASTER","slug":"iterator-address-is-less-than-log-headaddress-in-memory-scan-f49cb5","errorCode":null,"errorMessage":"Iterator address is less than log HeadAddress in memory-scan mode","messagePattern":"Iterator address is less than log HeadAddress in memory-scan mode","errorType":"exception","errorClass":"FasterException","httpStatus":null,"severity":"error","filePath":"cs/src/core/Allocator/VarLenBlittableScanIterator.cs","lineNumber":105,"sourceCode":"            recordInfo = default;\n\n            while (true)\n            {\n                currentAddress = nextAddress;\n                if (currentAddress >= endAddress)\n                    return false;\n\n                epoch?.Resume();\n                long headAddress = hlog.HeadAddress;\n\n                if (currentAddress < hlog.BeginAddress && !forceInMemory)\n                    currentAddress = hlog.BeginAddress;\n\n                // If currentAddress < headAddress and we're not buffering and not guaranteeing the records are in memory, fail.\n                if (frameSize == 0 && currentAddress < headAddress && !forceInMemory)\n                {\n                    epoch?.Suspend();\n                    throw new FasterException(\"Iterator address is less than log HeadAddress in memory-scan mode\");\n                }\n\n                var currentPage = currentAddress >> hlog.LogPageSizeBits;\n                var offset = currentAddress & hlog.PageSizeMask;\n\n                if (currentAddress < headAddress && !forceInMemory)\n                    BufferAndLoad(currentAddress, currentPage, currentPage % frameSize, headAddress, endAddress);\n\n                long physicalAddress = GetPhysicalAddress(currentAddress, headAddress, currentPage, offset);\n                int recordSize = hlog.GetRecordSize(physicalAddress).Item2;\n\n                // If record does not fit on page, skip to the next page.\n                if ((currentAddress & hlog.PageSizeMask) + recordSize > hlog.PageSize)\n                {\n                    nextAddress = (1 + (currentAddress >> hlog.LogPageSizeBits)) << hlog.LogPageSizeBits;\n                    epoch?.Suspend();\n                    continue;\n                }","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cs/src/core/Allocator/VarLenBlittableScanIterator.cs#L87-L123","documentation":"In memory-scan mode (frameSize == 0) the VarLenBlittableScanIterator keeps no on-disk page cache, so it can only return records currently resident in the log's in-memory region. If the iterator's currentAddress is below the log's HeadAddress (records have been truncated/evicted) and the caller has not passed forceInMemory, GetNext suspends the epoch and throws FasterException. It protects the caller from reading garbage or stale memory for records that no longer exist.","triggerScenarios":"Creating a scan iterator with an address older than the current HeadAddress on a log with no page buffering (frameSize 0) and calling GetNext, while forceInMemory is false. Happens when a long-running iterator lags behind while the log truncates its head, or when scanning from BeginAddress after the log has overwritten that region.","commonSituations":"A consumer iterating slowly while heavy upserts shift the HeadAddress past it; restarting a scan from a stale saved address; scanning BeginAddress on a long-running, memory-bounded log; configuring the blittable FasterLog with no scan buffering.","solutions":["Use ScanBufferingMode.DoublePageBuffering (frameSize > 0) so the iterator can buffer pages from disk instead of pure in-memory scan.","Pass forceInMemory: true only if you accept that records below HeadAddress cannot be returned and you just want to skip to the head.","Restart the scan from the current log.HeadAddress instead of the stale address.","If an iterator must survive truncation, persist and resume from addresses >= HeadAddress, or checkpoint the iterator address regularly.","Keep consumer throughput ahead of log truncation (periodic TruncateUntilHeadAddress-aware bookkeeping)."],"exampleFix":"// before: in-memory scan from a possibly stale address\nvar iter = new VarLenBlittableScanIterator<BlittableAllocator<..., ...>>(hlog, epoch, BeginAddress, long.MaxValue, frameSize: 0, headAddress, forceInMemory: false);\n// after: buffer pages from disk so addresses below head are readable\nvar iter = new VarLenBlittableScanIterator<...>(hlog, epoch, BeginAddress, long.MaxValue, frameSize: 2 * hlog.PageSize, headAddress, forceInMemory: false);\n// or, if in-memory only is intended:\n// var iter = new VarLenBlittableScanIterator<...>(hlog, epoch, Math.Max(BeginAddress, hlog.HeadAddress), long.MaxValue, 0, hlog.HeadAddress, forceInMemory: true);","handlingStrategy":"validation","validationCode":"// before creating an in-memory (frameSize 0) iterator, clamp the start address\nlong safeStart = Math.Max(startAddress, hlog.HeadAddress);\nif (startAddress < hlog.HeadAddress)\n    // either buffer pages from disk (frameSize > 0) or accept skipping to head\n    startAddress = safeStart;","typeGuard":null,"tryCatchPattern":"try {\n    while (iter.GetNext(out RecordInfo info)) { /* process */ }\n} catch (FasterException ex) when (ex.Message.Contains(\"less than log HeadAddress\")) {\n    // restart from current head or enable page buffering\n    iter.Dispose();\n    iter = hlog.Scan(hlog.HeadAddress, long.MaxValue, ScanBufferingMode.DoublePageBuffering, ...);\n}","preventionTips":["Persist/resume iterator addresses only if they are >= log.HeadAddress","Use DoublePageBuffering for any scan that may lag behind truncation","Monitor HeadAddress advance rate vs consumer speed","Pass forceInMemory:true only when skipping truncated records is acceptable"],"tags":["scan-iterator","address-out-of-range","truncation","faster-log"],"backgroundTag":"value-out-of-range","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}