microsoft/FASTER · error · Exception

Out of order message within session

Error message

Out of order message within session

What it means

FASTER's client session processes reply batches from the server in strict sequence-number order. Each incoming batch carries a BatchHeader whose SeqNo must equal lastSeqNo + 1; if it does not, the client's reply stream is corrupted or messages were dropped/reordered by the network or server, so the library throws to prevent silently applying out-of-order responses. Subscription sessions are exempt because their delivery order is inherently asynchronous.

Solutions

1. Ensure one ClientSession instance is used by exactly one thread (or serialized externally) at a time; replies for a session are not safe to process concurrently.
2. Recreate the session (and reconnect) after any network disruption so sequence numbers on both ends restart in sync.
3. Check the server deployment for restarts or proxies that can reorder/drop TCP frames; FASTER assumes reliable in-order TCP delivery.
4. If you maintain a fork, verify the server increments SeqNo per message exactly as the client's BatchHeader expects.

Example fix

// before: shared session across threads
clientSession.Read(...); // thread A
clientSession.Read(...); // thread B

// after: one session per logical consumer, or lock around use
lock (sessionLock) { clientSession.Read(...); }

// verify single-threaded use and healthy connection before reading
if (sessionDisposed || !socketConnected) { reconnectAndRecreateSession(); }

try { session.ProcessRepliesAndWait(...); }
catch (Exception ex) when (ex.Message == "Out of order message within session")
{
    // sequence stream corrupted: discard session, reconnect, rebuild session
    session.Dispose(); session = CreateNewSession();
}

When it happens

Trigger: ProcessReplies receives a batch whose BatchHeader.SeqNo is not lastSeqNo + 1 on a non-subscription session — typically after a dropped TCP packet boundary mismatch, a server restart mid-stream, or concurrent use of the same ClientSession from multiple threads interleaving reads.

Common situations: Sharing a single ClientSESSION across threads without synchronization; reconnecting a socket while the server still holds a different sequence state; a custom network proxy/relay reordering frames; running against a modified server build that resets or skips sequence numbers.

Related errors

• Unexpected status of SubscribeKV
• Failed to connect server.
• Cannot use BlittableParameterSerializer with non-blittable…
• The inner list is full!
• The list is empty!

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/3283922aaff6184b. Report an issue: GitHub.

Appendix: source

Thrown at cs/remote/src/FASTER.client/ClientSession.cs:347

            Dispose();
        }


        int lastSeqNo = -1;
        readonly Dictionary<int, (Key, Value, Context)> pubsubPendingContext = new();
        readonly Dictionary<int, (Key, Input, Output, Context)> readRmwPendingContext = new();
        readonly Dictionary<int, TaskCompletionSource<(Status, Output)>> readRmwPendingTcs = new();

        internal void ProcessReplies(byte[] buf, int offset)
        {
            Output defaultOutput = default;
            fixed (byte* b = &buf[offset])
            {
                var src = b;
                var seqNo = ((BatchHeader*)src)->SeqNo;
                var count = ((BatchHeader*)src)->NumMessages;
                if (seqNo != lastSeqNo + 1 && !subscriptionSession)
                    throw new Exception("Out of order message within session");
                lastSeqNo = seqNo;

                src += BatchHeader.Size;

                for (int i = 0; i < count; i++)
                {
                    switch ((MessageType)(*src++))
                    {
                        case MessageType.Upsert:
                            {
                                var status = ReadStatus(ref src);
                                (Key, Value, Context) result = upsertQueue.Dequeue();
                                functions.UpsertCompletionCallback(ref result.Item1, ref result.Item2, result.Item3);
                                break;
                            }
                        case MessageType.UpsertAsync:
                            {
                                var status = ReadStatus(ref src);

View on GitHub (pinned to 321d872eab)