microsoft/garnet · warning · GarnetException

Failed to validate main store metadata at insertion

Error message

Failed to validate main store metadata at insertion

What it means

Raised inside AddCheckpointEntry.ValidateCheckpointEntry when replicationManager.TryAcquireSettledMetadataForMainStore fails for the entry being inserted. Crucially it is thrown inside a local try/catch that logs the message at Error level and returns false, so the exception does not propagate to the caller; the visible symptom is a logged error and a failed validation rather than a thrown GarnetException.

Source

Thrown at libs/cluster/Server/Replication/CheckpointStore.cs:148

            else
            {
                // We don't have multiple writers because this method is called under the CheckpointLock
                // So it is safe to update in-place.
                tail.next = entry;
                tail = tail.next;
            }

            logger?.LogCheckpointEntry(LogLevel.Trace, nameof(AddCheckpointEntry), entry);

            if (safelyRemoveOutdated)
                DeleteOutdatedCheckpoints();

            bool ValidateCheckpointEntry(CheckpointEntry entry)
            {
                try
                {
                    if (!clusterProvider.replicationManager.TryAcquireSettledMetadataForMainStore(entry, out _, out _))
                        throw new GarnetException("Failed to validate main store metadata at insertion");

                    return true;
                }
                catch (Exception ex)
                {
                    logger?.LogCheckpointEntry(LogLevel.Error, ex.Message, entry);
                    return false;
                }
            }
        }

        /// <summary>
        /// Method used to delete outdated checkpoints from in-memory list of checkpoint entries
        /// </summary>
        private void DeleteOutdatedCheckpoints()
        {
            // If only one in-memory checkpoint return
            if (head == tail) return;

View on GitHub (pinned to 951b0fc683)

Solutions

  1. Check logs for this Error-level message and inspect whether the replication manager was healthy at insertion time.
  2. Ensure checkpoint insertion happens after main-store metadata has settled (replication manager fully initialized).
  3. If it recurs, verify TryAcquireSettledMetadataForMainStore's preconditions (store open, metadata present) are met before AddCheckpointEntry.
Defensive patterns

Strategy: validation

Validate before calling

// Ensure replication manager is ready before checkpoint insertion
if (!clusterProvider.replicationManager.TryAcquireSettledMetadataForMainStore(entry, out _, out _))
    logger?.LogError("Skipping checkpoint insertion: main-store metadata not settled for {entry}", entry);
else
    store.AddCheckpointEntry(entry, fullCheckpoint);

Try / catch

// The exception is already caught internally and logged; monitor logs rather than catching
if (!added)
{
    logger?.LogWarning("Checkpoint entry validation failed; investigate replication manager state");
}

Prevention

When it happens

Trigger: Inserting a checkpoint entry whose main-store settled metadata cannot be acquired/verified by the replication manager (e.g. metadata not yet settled, store not initialized, replication manager stopped).

Common situations: Checkpoint completion racing with replication shutdown; a replica attempting checkpoint insertion before its main-store metadata has settled; a corrupted or missing metadata token after recovery.

Related errors

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/353f8213bbbf9d43. Report an issue: GitHub.