dotnet/orleans · error · InvalidOperationException

Recovered {name} does not match. Written: {Serialize(written

Error message

Recovered {name} does not match. Written: {Serialize(written)}, recovered: {Serialize(recovered)}.

What it means

Thrown by the JournalingAzureBlobJson sample's EnsureEqual<T> helper after it rehydrates a journaled grain and compares the recovered durable state against what was originally written. The sample exercises every IDurable* collection (dictionary, list, queue, set, value) plus persistent state, writes it through an Azure append-blob journal, restarts, and asserts the recovered snapshot is bit-for-bit equal. A mismatch means the journaling codec, ordering, or storage layer did not round-trip the data correctly.

Source

Thrown at samples/JournalingAzureBlobJson/JournalingAzureBlobJson/Program.cs:140

        if (written.Count != recovered.Count)
        {
            throw new InvalidOperationException($"Recovered {name} count does not match. Written: {written.Count}, recovered: {recovered.Count}.");
        }

        for (var i = 0; i < written.Count; i++)
        {
            if (!equals(written[i], recovered[i]))
            {
                throw new InvalidOperationException($"Recovered {name} item {i} does not match. Written: {Serialize(written[i])}, recovered: {Serialize(recovered[i])}.");
            }
        }
    }

    private static void EnsureEqual<T>(string name, T written, T recovered, Func<T, T, bool> equals)
    {
        if (!equals(written, recovered))
        {
            throw new InvalidOperationException($"Recovered {name} does not match. Written: {Serialize(written)}, recovered: {Serialize(recovered)}.");
        }
    }

    private static bool InventoryEntryEquals(InventoryEntry left, InventoryEntry right)
        => string.Equals(left.Key, right.Key, StringComparison.Ordinal)
            && InventoryItemEquals(left.Value, right.Value);

    private static bool InventoryItemEquals(InventoryItem left, InventoryItem right)
        => string.Equals(left.Sku, right.Sku, StringComparison.Ordinal)
            && left.Quantity == right.Quantity
            && left.UnitPrice == right.UnitPrice
            && left.Attributes.SequenceEqual(right.Attributes, StringComparer.Ordinal);

    private static bool JournalEventEquals(JournalEvent left, JournalEvent right)
        => string.Equals(left.EventId, right.EventId, StringComparison.Ordinal)
            && left.Timestamp == right.Timestamp
            && string.Equals(left.Kind, right.Kind, StringComparison.Ordinal)
            && left.Notes.SequenceEqual(right.Notes, StringComparer.Ordinal);

View on GitHub (pinned to fca799fa70)

Solutions

1. Re-run with --reset (the default) so the sample wipes and rewrites the append-blob journal before comparing, removing stale-data mismatches.
2. Inspect the thrown message: it prints Serialize(written) and Serialize(recovered); diff those JSON payloads to find exactly which field diverged.
3. If you modified InventoryItem/JournalEvent/WorkItem records, make sure [GenerateSerializer] IDs are stable and no [Id] was reordered or removed — changed serializer IDs corrupt recovery.
4. Ensure the grain calls WriteStateAsync and the second activation is a true fresh activation (DeactivateOnIdle / new silo) so recovery actually reloads from the blob.

Example fix

// before: stale journal data survives across schema changes
// run with --no-reset against an old blob
var resetBlob = !args.Contains("--no-reset", StringComparer.OrdinalIgnoreCase);

// after: force a clean journal rewrite when the schema changed
dotnet run -- --connection-string ... --reset

// Before reading the summary, confirm the scenario ran and balance was written
if (balance.Value is null || inventory.Count == 0)
{
    Console.WriteLine("Scenario has not been run yet; call RunScenario first.");
    return;
}

null

null

When it happens

Trigger: EnsureEqual<T>(name, written, recovered, equals) is invoked when equals(written, recovered) returns false; the per-item overload at line ~140 fires on the first index i where the custom comparer (e.g. InventoryItemEquals, which compares Sku, Quantity, UnitPrice and Attributes.SequenceEqual with an Ordinal comparer) detects a divergence. This runs at the tail end of the sample's recovery phase, after WriteStateAsync on the grain followed by a fresh activation read.

Common situations: Editing the sample's grain so the durable state shape changes without re-clearing the blob (stale journal data left from a previous schema). Running against a blob that another sample/activation wrote with different data. A custom serializer or GrainStorageSerializer that is non-idempotent or drops fields. Attribute ordering differences when the comparer is switched away from StringComparer.Ordinal.

Related errors

• The grain did not reactivate after DeactivateOnIdle.
• Recovered {name} count does not match. Written: {written.Cou
• Recovered {name} item {i} does not match. Written: {Serializ
• Balance has not been written.
• Missing value for {name}.

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/bc09de5d35f32383. Report an issue: GitHub.