microsoft/FASTER · critical · FasterException

Unable to find valid HybridLog token

Error message

Unable to find valid HybridLog token

What it means

During recovery, FASTER locates the HybridLog checkpoint closest to the requested recovery version via GetClosestHybridLogCheckpointInfo. If no HybridLog checkpoint information can be found (recoveredHlcInfo is default), there is no valid log token to recover from, and this FasterException is thrown. It means the checkpoint store contains no usable hybrid log token for the requested recovery point.

Solutions

  1. List available checkpoints via checkpointManager and recover to a token/version that actually has a HybridLog checkpoint.
  2. Take a full (or hybrid log) checkpoint with TakeFullCheckpoint or TakeHybridLogCheckpoint before attempting recovery.
  3. Verify the checkpoint directory and checkpointManager configuration point to the store that produced the token.
  4. If recovering to a specific version, ensure the checkpoint was taken with incremental snapshot support (fullSnapshot=false) so delta logs exist.

Example fix

// before
faster.Recover(42); // fails: no hybrid log checkpoint covers version 42
// after
var (ok, token) = await faster.TakeFullCheckpointAsync();
await checkpointManager.FinalizeCheckpoint(...);
faster.Recover(token); // recover to an existing token
Defensive patterns

Strategy: validation

Validate before calling

// Before recovery, verify a hybrid log checkpoint exists for the target version
var available = checkpointManager.GetRecoverableHybridLogCheckpoints();
if (available.Count == 0)
    throw new InvalidOperationException("No hybrid log checkpoints available; take a full checkpoint first.");

Try / catch

try
{
    await faster.RecoverAsync(recoverVersion);
}
catch (FasterException ex) when (ex.Message.Contains("Unable to find valid HybridLog token"))
{
    logger.LogWarning("No valid hybrid log token; recovering latest or retaking checkpoint");
    await faster.TakeFullCheckpointAsync();
}

Prevention

When it happens

Trigger: Calling Recover(version)/RecoverAsync(version) on a FasterLog or FASTER instance when no HybridLog checkpoint exists in the checkpoint directory or checkpoint manager, when the hybrid log checkpoint files were deleted or partially written, or when recovering by version/index in a store that only ever took index-only checkpoints.

Common situations: Pointing recovery at the wrong checkpoint folder, running recovery before any TakeFullCheckpoint/TakeHybridLogCheckpoint completed, crash during checkpointing leaving an incomplete token entry, or a version/index passed that predates every available hybrid log checkpoint.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/5f811e76d9f5628a. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/core/Index/Recovery/Recovery.cs:322

                    continue;
                }

                logger?.LogInformation("Index Checkpoint: {indexToken}", indexToken);
                recoveredICInfo.info.DebugPrint(logger);
                closestToken = indexToken;
                break;
            }
        }

        private void FindRecoveryInfo(long requestedVersion, out HybridLogCheckpointInfo recoveredHlcInfo,
            out IndexCheckpointInfo recoveredICInfo)
        {
            logger?.LogInformation("********* Primary Recovery Information ********");

            GetClosestHybridLogCheckpointInfo(requestedVersion, out var closestToken, out recoveredHlcInfo, out recoveredCommitCookie);

            if (recoveredHlcInfo.IsDefault())
                throw new FasterException("Unable to find valid HybridLog token");

            if (recoveredHlcInfo.deltaLog != null)
            {
                recoveredHlcInfo.Dispose();
                // need to actually scan delta log now
                recoveredHlcInfo.Recover(closestToken, checkpointManager, hlog.LogPageSizeBits, out _, true);
            }
            recoveredHlcInfo.info.DebugPrint(logger);

            GetClosestIndexCheckpointInfo(ref recoveredHlcInfo, out _, out recoveredICInfo);

            if (recoveredICInfo.IsDefault())
            {
                logger?.LogInformation("No index checkpoint found, recovering from beginning of log");
            }
        }

        private static bool IsCompatible(in IndexRecoveryInfo indexInfo, in HybridLogRecoveryInfo recoveryInfo)

View on GitHub (pinned to 321d872eab)