What causes "Cannot recover from (" in microsoft/FASTER?

Always pass the index and hybrid log tokens captured together from the same TakeFullCheckpoint/TakeIndexCheckpoint+TakeHybridLogCheckpoint call.

microsoft/FASTER · critical · FasterException

Cannot recover from (

Error message

Cannot recover from (

What it means

GetRecoveryInfo validates that the index checkpoint and hybrid log checkpoint form a compatible pair via IsCompatible before recovery proceeds. When the two Guid tokens come from different checkpoint sessions (their internal checkpoint metadata do not match), recovery cannot combine them and this exception is thrown. It guards against assembling a corrupted logical state from unrelated snapshots.

Solutions

Always pass the index and hybrid log tokens captured together from the same TakeFullCheckpoint/TakeIndexCheckpoint+TakeHybridLogCheckpoint call.
Use the secondary checkpoint API (Recover with a single token / checkpoint entry) so both tokens are resolved from one checkpoint entry.
Verify the checkpoint directory contents: both index and hybrid log checkpoint metadata for the pair must exist and come from the same session.
Retake a full checkpoint to obtain a fresh consistent token pair.

Example fix

// before
faster.Recover(oldIndexToken, newHybridLogToken); // tokens from different sessions
// after
var token = await faster.TakeFullCheckpointAsync(); // capture both tokens together
SaveBothTokens(token.indexToken, token.hlogToken);
faster.Recover(token.indexToken, token.hlogToken);

Defensive patterns

Strategy: validation

Validate before calling

// Persist and read index + hybrid log tokens as one atomic record
var (indexToken, hlogToken) = LoadTokenPair(); // always saved together at checkpoint time
if (indexToken == Guid.Empty || hlogToken == Guid.Empty)
    throw new InvalidOperationException("Incomplete token pair; retake checkpoint.");
faster.Recover(indexToken, hlogToken);

Try / catch

try
{
    faster.Recover(indexToken, hybridLogToken);
}
catch (FasterException ex) when (ex.Message.StartsWith("Cannot recover from ("))
{
    logger.LogError("Incompatible checkpoint pair {A}/{B}; falling back to latest checkpoint", indexToken, hybridLogToken);
    faster.Recover();
}

Prevention

Store index and hybrid log tokens together in one transactional record.
Use TakeFullCheckpoint so a single call yields a consistent pair.
Never hand-copy tokens between environments or runs.

When it happens

Trigger: Calling Recover(indexToken, hybridLogToken) with tokens taken at different times or from different FASTER instances, or manually specifying one token from an old checkpoint and another from a newer one.

Common situations: Persisting index and hybrid log tokens separately (e.g. in different config stores) and one being updated while the other was not; copying checkpoint directories selectively; testing with mixed tokens from multiple runs.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/56020ab13aa109fe. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/core/Index/Recovery/Recovery.cs:397

                    recoveredICInfo.Recover(indexToken, checkpointManager);
                    recoveredICInfo.info.DebugPrint(logger);
                }
            }
            catch
            {
                recoveredICInfo = default;
            }

            if (recoveredICInfo.IsDefault())
            {
                logger?.LogInformation("Invalid index checkpoint token, recovering from beginning of log");
            }
            else
            {
                // Check if the two checkpoints are compatible for recovery
                if (!IsCompatible(recoveredICInfo.info, recoveredHLCInfo.info))
                {
                    throw new FasterException("Cannot recover from (" + indexToken.ToString() + "," + hybridLogToken.ToString() + ") checkpoint pair!\n");
                }
            }
        }

        /// <inheritdoc />
        public void Reset()
        {
            // Reset the hash index
            Array.Clear(state[resizeInfo.version].tableRaw, 0, state[resizeInfo.version].tableRaw.Length);
            overflowBucketsAllocator.Dispose();
            overflowBucketsAllocator = new MallocFixedPageSize<HashBucket>();

            // Reset the hybrid log
            hlog.Reset();
        }


        private long InternalRecover(IndexCheckpointInfo recoveredICInfo, HybridLogCheckpointInfo recoveredHLCInfo, int numPagesToPreload, bool undoNextVersion, long recoverTo)

View on GitHub (pinned to 321d872eab)