What causes "One or more errors occurred." in dotnet/orleans?

Inspect AggregateException.InnerExceptions to identify all root causes — the first may not be the most important.

dotnet/orleans · error · AggregateException

One or more errors occurred.

Error message

One or more errors occurred.

What it means

This AggregateException is thrown by EventHubAdapterReceiver.Shutdown when multiple independent components fail during shutdown — the method collects exceptions from checkpoint flushing, receiver closing, cache disposal, and the close-task await into a list, and if more than one exception is collected it wraps them all in an AggregateException rather than reporting only the first. If exactly one exception occurs it is re-thrown directly via ExceptionDispatchInfo preserving the original stack trace.

Source

Thrown at src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:367

                this.monitor?.TrackShutdown(true, watch.Elapsed, null);
            }
            catch (Exception ex)
            {
                watch.Stop();
                this.monitor?.TrackShutdown(false, watch.Elapsed, ex);
                throw;
            }

            static void ThrowIfAny(List<Exception> exceptions)
            {
                if (exceptions.Count == 1)
                {
                    ExceptionDispatchInfo.Capture(exceptions[0]).Throw();
                }

                if (exceptions.Count > 1)
                {
                    throw new AggregateException(exceptions);
                }
            }
        }

        private static IEventHubReceiver CreateReceiver(EventHubPartitionSettings partitionSettings, string offset, ILogger logger)
        {
            return new EventHubReceiverProxy(partitionSettings, offset, logger);
        }

        /// <summary>
        /// For test purpose. ConfigureDataGeneratorForStream will configure a data generator for the stream
        /// </summary>
        /// <param name="streamId"></param>
        internal void ConfigureDataGeneratorForStream(StreamId streamId)
        {
            (this.receiver as EventHubPartitionGeneratorReceiver)?.ConfigureDataGeneratorForStream(streamId);
        }

View on GitHub (pinned to fca799fa70)

Solutions

Inspect AggregateException.InnerExceptions to identify all root causes — the first may not be the most important.
Ensure Azure Storage (for checkpointing) and Event Hub connectivity are healthy before initiating a graceful shutdown.
If using a timeout for Shutdown, ensure it is generous enough (e.g., 30+ seconds) to allow checkpoint flushing and receiver closing to complete.
If the silo is shutting down due to an error, resolve the underlying connectivity issue first, then retry shutdown.

Example fix

// before
try
{
    await receiver.Shutdown(TimeSpan.FromSeconds(5));
}
catch (Exception ex)
{
    logger.LogError(ex, "Shutdown failed"); // loses inner exceptions
}

// after
try
{
    await receiver.Shutdown(TimeSpan.FromSeconds(30));
}
catch (AggregateException aggEx)
{
    foreach (var inner in aggEx.InnerExceptions)
        logger.LogError(inner, "Shutdown sub-operation failed");
}
catch (Exception ex)
{
    logger.LogError(ex, "Shutdown failed");
}

Defensive patterns

Strategy: try-catch

Try / catch

try
{
    await receiver.Shutdown(shutdownTimeout);
}
catch (AggregateException aggEx)
{
    // Log all inner exceptions — each represents a different failed sub-operation
    foreach (var inner in aggEx.InnerExceptions)
    {
        logger.LogWarning(inner, "Shutdown sub-operation failed for partition {Partition}", partition);
    }
    // Decide whether to rethrow based on your error policy
}
catch (Exception ex) when (ex is not AggregateException)
{
    // Single exception — re-thrown via ExceptionDispatchInfo, preserves original stack
    logger.LogError(ex, "Shutdown failed for partition {Partition}", partition);
    throw;
}

Prevention

Use a generous shutdown timeout (30+ seconds) to allow checkpoint flushing and receiver closing to complete under load.
Ensure Azure Storage (checkpointing) and Event Hub connectivity are healthy before graceful shutdown.
Monitor for partial connectivity issues (e.g., storage throttling) that can cause cascading shutdown failures.
Always catch AggregateException and inspect InnerExceptions, not just the outer Message.

When it happens

Trigger: Calling Shutdown (or the silo stopping and triggering receiver shutdown) when two or more of the following fail simultaneously: (1) checkpointer.FlushAsync throws (e.g., Azure Storage outage), (2) EventHubReceiver.CloseAsync throws (e.g., network failure), (3) cache.Dispose throws (e.g., eviction strategy disposal error), (4) awaiting the close task throws due to timeout or cancellation.

Common situations: Silo shutdown during a partial network outage where both Event Hub connections and checkpoint storage (Azure Blob Storage) are unreachable; a timeout is passed to Shutdown that is too short, causing the CancellationTokenSource to cancel both the flush and the close operations; cascading failures where a connection drop affects both the Event Hub receiver and the storage-based checkpointer.

Related errors

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/a0cdcfa8017d3423. Report an issue: GitHub.