hyperledger/fabric · warning

too many requests for %s, exceeding concurrency limit (%d)

Error message

too many requests for %s, exceeding concurrency limit (%d)

What it means

A unary gRPC server interceptor on the peer enforces a per-service concurrency limit using a semaphore. When a request arrives and no semaphore slot is available (TryAcquire fails), the interceptor logs and rejects the call with this error instead of invoking the handler, protecting the peer from overload.

Source

Thrown at internal/peer/node/grpc_limiters.go:53

	}
	if gatewayConcurrency != 0 {
		logger.Infof("concurrency limit for gateway service is %d", gatewayConcurrency)
		semaphores["/gateway.Gateway"] = semaphore.New(gatewayConcurrency)
	}

	return semaphores
}

func unaryGrpcLimiter(semaphores map[string]semaphore.Semaphore) grpc.UnaryServerInterceptor {
	return func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
		serviceName := getServiceName(info.FullMethod)
		sema, ok := semaphores[serviceName]
		if !ok {
			return handler(ctx, req)
		}
		if !sema.TryAcquire() {
			logger.Errorf("Too many requests for %s, exceeding concurrency limit (%d)", serviceName, cap(sema))
			return nil, errors.Errorf("too many requests for %s, exceeding concurrency limit (%d)", serviceName, cap(sema))
		}
		defer sema.Release()
		return handler(ctx, req)
	}
}

func streamGrpcLimiter(semaphores map[string]semaphore.Semaphore) grpc.StreamServerInterceptor {
	return func(srv any, ss grpc.ServerStream, info *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
		serviceName := getServiceName(info.FullMethod)
		sema, ok := semaphores[serviceName]
		if !ok {
			return handler(srv, ss)
		}
		if !sema.TryAcquire() {
			logger.Errorf("Too many requests for %s, exceeding concurrency limit (%d)", serviceName, cap(sema))
			return errors.Errorf("too many requests for %s, exceeding concurrency limit (%d)", serviceName, cap(sema))
		}
		defer sema.Release()

View on GitHub (pinned to 2736b63f8f)

Solutions

1. Retry the request with backoff; the error is transient and clears when in-flight requests finish.
2. Increase the peer's gRPC concurrency limit configuration for that service if load is legitimately high.
3. Reduce client concurrency / add client-side rate limiting or load balancing across multiple peers.

// client-side: cap concurrent in-flight unary RPCs below the peer's configured limiter
sem := make(chan struct{}, maxClientConcurrency)
sem <- struct{}{}
defer func() { <-sem }()

for attempt := 0; attempt < 5; attempt++ {
    _, err := client.Invoke(ctx, req)
    if err != nil && strings.Contains(err.Error(), "exceeding concurrency limit") {
        time.Sleep(backoff(attempt))
        continue
    }
    return err
}

When it happens

Trigger: Sending a unary RPC to a peer service (e.g. endorsement, deliver) whose configured limiter semaphore is already at capacity with concurrent in-flight requests.

Common situations: Load spikes or benchmarking against the peer without tuning vm/chaincode or deliver concurrency limits; many SDK clients hammering the peer simultaneously; leaking streams that keep concurrency slots occupied.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• orderer `%s` hung up without sending status
• chaincode already successfully installed (package ID '%s')
• a ledger is being created from a snapshot at %s. Call ledger
• cannot find ledger for channel %s
• as another peer node command is executing, wait for that com

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/7cbc25e55ca9d1bd. Report an issue: GitHub.