dotnet/orleans · error · InvalidOperationException

Cached Event Hub message is missing its offset.

Error message

Cached Event Hub message is missing its offset.

What it means

EventHubDataAdapter.GetOffset throws InvalidOperationException when SegmentBuilder.ReadNextString returns null while reading the offset from a CachedMessage's Segment. The offset is the Event Hub message's OffsetString — a string identifier for the event's position in the partition. A null return means the segment data is corrupted, was not properly encoded, or the segment has been partially overwritten. This method is called during cache purge (OnPurge) to update the checkpointer with the last purged offset.

Source

Thrown at src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubDataAdapter.cs:90

            };
        }

        public virtual StreamPosition GetStreamPosition(string partition, EventData queueMessage)
        {
            StreamId streamId = this.GetStreamIdentity(queueMessage);
            StreamSequenceToken token =
                new EventHubSequenceTokenV2(queueMessage.OffsetString, queueMessage.SequenceNumber, 0);
            return new StreamPosition(streamId, token);
        }

        /// <summary>
        /// Get offset from cached message.  Left to derived class, as only it knows how to get this from the cached message.
        /// </summary>
        public virtual string GetOffset(CachedMessage lastItemPurged)
        {
            int readOffset = 0;
            return SegmentBuilder.ReadNextString(lastItemPurged.Segment, ref readOffset)
                ?? throw new InvalidOperationException("Cached Event Hub message is missing its offset.");
        }

        /// <summary>
        /// Get the Event Hub partition key to use for a stream.
        /// </summary>
        /// <param name="streamId">The stream Guid.</param>
        /// <returns>The partition key to use for the stream.</returns>
        public virtual string GetPartitionKey(StreamId streamId) => streamId.GetKeyAsString();

        /// <summary>
        /// Get the <see cref="IStreamIdentity"/> for an event message.
        /// </summary>
        /// <param name="queueMessage">The event message.</param>
        /// <returns>The stream identity.</returns>
        public virtual StreamId GetStreamIdentity(EventData queueMessage)
        {
            string streamKey = queueMessage.PartitionKey;
            string? streamNamespace = queueMessage.GetStreamNamespaceProperty();

View on GitHub (pinned to fca799fa70)

Solutions

  1. If using a custom EventHubDataAdapter subclass, ensure EncodeMessageIntoSegment and GetOffset agree on the segment layout — the first field written must be the offset string.
  2. Clear the cache and checkpoint state (Azure Blob Storage lease/checkpoint blob) after upgrading Orleans if the CachedMessage segment format changed, to avoid loading stale incompatible messages.
  3. If the error recurs, check for a custom IEventHubDataAdapter or a custom IEventHubQueueCache that may alter segment encoding.
  4. Inspect the EventHubDataAdapter.EncodeMessageIntoSegment to confirm offset is the first SegmentBuilder.Append call, matching the read order in GetOffset.

Example fix

// before — custom data adapter overrides encoding but forgets offset
public class CustomDataAdapter : EventHubDataAdapter
{
    protected override ArraySegment<byte> EncodeMessageIntoSegment(
        EventData msg, Func<int, ArraySegment<byte>> getSegment)
    {
        // offset NOT written first — GetOffset will fail
        ...
    }
}

// after — keep offset as first field, or override GetOffset	public class CustomDataAdapter : EventHubDataAdapter
{
    protected override ArraySegment<byte> EncodeMessageIntoSegment(
        EventData msg, Func<int, ArraySegment<byte>> getSegment)
    {
        // write offset FIRST to match base GetOffset
        ...
    }
    // or override GetOffset to match your layout
    public override string GetOffset(CachedMessage lastItemPurged)
    {
        int readOffset = yourCustomOffsetPosition;
        return SegmentBuilder.ReadNextString(lastItemPurged.Segment, ref readOffset)
            ?? throw new InvalidOperationException("offset missing");
    }
}
Defensive patterns

Strategy: try-catch

Validate before calling

// If using a custom data adapter, validate segment encoding matches GetOffset:
// Ensure offset is the FIRST field written in EncodeMessageIntoSegment.

Try / catch

// When handling cache purge in custom infrastructure:
try
{
    var offset = dataAdapter.GetOffset(lastItemPurged);
    checkpointer.Update(offset, DateTime.UtcNow, CancellationToken.None);
}
catch (InvalidOperationException ex) when (ex.Message.Contains("missing its offset"))
{
    logger.LogError(ex, "Cached message segment is corrupted; skipping checkpoint update");
    // Do not rethrow in purge path — one bad message should not crash the silo
}

Prevention

When it happens

Trigger: During cache eviction when OnPurge calls dataAdapter.GetOffset(lastItemPurged) and the CachedMessage's Segment does not contain a valid encoded string at the expected position. This can happen if EncodeMessageIntoSegment wrote data in a different order than GetOffset reads it, if the segment buffer was corrupted or reused, or if a custom IEventHubDataAdapter subclass overrides encoding but not GetOffset (or vice versa).

Common situations: A custom EventHubDataAdapter subclass that overrides EncodeMessageIntoSegment to change the segment layout but does not also override GetOffset to match; memory corruption from a buffer pool bug where a FixedSizeBuffer segment is overwritten; a version mismatch where cached messages from an older Orleans version are loaded by a newer data adapter with a different segment format; extremely rare race conditions in the eviction strategy.

Related errors

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/ff125e70286fb357. Report an issue: GitHub.