microsoft/FASTER · error · FasterException

Iterator address is less than log HeadAddress in…

Error message

Iterator address is less than log HeadAddress in memory-scan mode

What it means

In memory-scan mode (frameSize == 0) the VarLenBlittableScanIterator keeps no on-disk page cache, so it can only return records currently resident in the log's in-memory region. If the iterator's currentAddress is below the log's HeadAddress (records have been truncated/evicted) and the caller has not passed forceInMemory, GetNext suspends the epoch and throws FasterException. It protects the caller from reading garbage or stale memory for records that no longer exist.

Solutions

1. Use ScanBufferingMode.DoublePageBuffering (frameSize > 0) so the iterator can buffer pages from disk instead of pure in-memory scan.
2. Pass forceInMemory: true only if you accept that records below HeadAddress cannot be returned and you just want to skip to the head.
3. Restart the scan from the current log.HeadAddress instead of the stale address.
4. If an iterator must survive truncation, persist and resume from addresses >= HeadAddress, or checkpoint the iterator address regularly.
5. Keep consumer throughput ahead of log truncation (periodic TruncateUntilHeadAddress-aware bookkeeping).

Example fix

// before: in-memory scan from a possibly stale address
var iter = new VarLenBlittableScanIterator<BlittableAllocator<..., ...>>(hlog, epoch, BeginAddress, long.MaxValue, frameSize: 0, headAddress, forceInMemory: false);
// after: buffer pages from disk so addresses below head are readable
var iter = new VarLenBlittableScanIterator<...>(hlog, epoch, BeginAddress, long.MaxValue, frameSize: 2 * hlog.PageSize, headAddress, forceInMemory: false);
// or, if in-memory only is intended:
// var iter = new VarLenBlittableScanIterator<...>(hlog, epoch, Math.Max(BeginAddress, hlog.HeadAddress), long.MaxValue, 0, hlog.HeadAddress, forceInMemory: true);

// before creating an in-memory (frameSize 0) iterator, clamp the start address
long safeStart = Math.Max(startAddress, hlog.HeadAddress);
if (startAddress < hlog.HeadAddress)
    // either buffer pages from disk (frameSize > 0) or accept skipping to head
    startAddress = safeStart;

try {
    while (iter.GetNext(out RecordInfo info)) { /* process */ }
} catch (FasterException ex) when (ex.Message.Contains("less than log HeadAddress")) {
    // restart from current head or enable page buffering
    iter.Dispose();
    iter = hlog.Scan(hlog.HeadAddress, long.MaxValue, ScanBufferingMode.DoublePageBuffering, ...);
}

When it happens

Trigger: Creating a scan iterator with an address older than the current HeadAddress on a log with no page buffering (frameSize 0) and calling GetNext, while forceInMemory is false. Happens when a long-running iterator lags behind while the log truncates its head, or when scanning from BeginAddress after the log has overwritten that region.

Common situations: A consumer iterating slowly while heavy upserts shift the HeadAddress past it; restarting a scan from a stale saved address; scanning BeginAddress on a long-running, memory-bounded log; configuring the blittable FasterLog with no scan buffering.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Page read from storage failed, skipping page. Inner…
• Use GetNext(out RecordInfo) to retrieve references to…

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/f49cb5eb26c1478c. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/core/Allocator/VarLenBlittableScanIterator.cs:105

            recordInfo = default;

            while (true)
            {
                currentAddress = nextAddress;
                if (currentAddress >= endAddress)
                    return false;

                epoch?.Resume();
                long headAddress = hlog.HeadAddress;

                if (currentAddress < hlog.BeginAddress && !forceInMemory)
                    currentAddress = hlog.BeginAddress;

                // If currentAddress < headAddress and we're not buffering and not guaranteeing the records are in memory, fail.
                if (frameSize == 0 && currentAddress < headAddress && !forceInMemory)
                {
                    epoch?.Suspend();
                    throw new FasterException("Iterator address is less than log HeadAddress in memory-scan mode");
                }

                var currentPage = currentAddress >> hlog.LogPageSizeBits;
                var offset = currentAddress & hlog.PageSizeMask;

                if (currentAddress < headAddress && !forceInMemory)
                    BufferAndLoad(currentAddress, currentPage, currentPage % frameSize, headAddress, endAddress);

                long physicalAddress = GetPhysicalAddress(currentAddress, headAddress, currentPage, offset);
                int recordSize = hlog.GetRecordSize(physicalAddress).Item2;

                // If record does not fit on page, skip to the next page.
                if ((currentAddress & hlog.PageSizeMask) + recordSize > hlog.PageSize)
                {
                    nextAddress = (1 + (currentAddress >> hlog.LogPageSizeBits)) << hlog.LogPageSizeBits;
                    epoch?.Suspend();
                    continue;
                }

View on GitHub (pinned to 321d872eab)