temporalio/temporal · error

internal error, reference-id: %v

Error message

internal error, reference-id: %v

What it means

logInternalError logs an internal error with full detail and returns a sanitized opaque error containing only a fresh UUID reference-id. The library throws this to hide internal (potentially sensitive) Nexus callback errors from end users while still allowing operators to correlate the user-visible message with the detailed server log entry.

Source

Thrown at chasm/lib/callback/invocable_internal.go:32

	callbackspb "go.temporal.io/server/chasm/lib/callback/gen/callbackpb/v1"
	"go.temporal.io/server/common/log"
	"go.temporal.io/server/common/log/tag"
	"go.temporal.io/server/common/namespace"
	commonnexus "go.temporal.io/server/common/nexus"
	"go.temporal.io/server/common/nexus/nexusrpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/types/known/timestamppb"
)

// logInternalError emits a log statement for internalMsg, tagged with both
// internalErr and a reference-id. An opaque error containing the reference-id is
// returned. Intended to be used to hide internal errors from end users.
func logInternalError(logger log.Logger, internalMsg string, internalErr error) error {
	referenceID := uuid.NewString()
	logger.Error(internalMsg, tag.Error(internalErr), tag.String("reference-id", referenceID))
	return fmt.Errorf("internal error, reference-id: %v", referenceID)
}

// invocableInternal is an invocable that delivers the Nexus operation completion data to History for cross-shard
// callbacks.
type invocableInternal struct {
	callback   *callbackspb.Callback_Nexus
	attempt    int32
	completion nexusrpc.CompleteOperationOptions
	requestID  string
}

func (c invocableInternal) WrapError(result invocationResult, err error) error {
	// Return the invocation result error if present
	if resultErr := result.error(); resultErr != nil {
		return resultErr
	}

	return err

View on GitHub (pinned to bde624efd1)

Solutions

1. Copy the reference-id and search the history/worker service logs for it to find the underlying internalErr with full detail
2. Verify history service is healthy and reachable from the service where the Nexus callback was invoked
3. Check Nexus operation callback data integrity; re-trigger the callback if a transient history error occurred
4. If reproducible, file a bug with the reference-id and logs — most internal errors indicate a real defect

// Go
err := invocable.Invoke(ctx)
if err != nil && strings.HasPrefix(err.Error(), "internal error, reference-id:") {
    refID := strings.TrimPrefix(err.Error(), "internal error, reference-id: ")
    logger.Warn("nexus callback failed internally", "reference-id", refID)
    // surface refID to user; look it up in server logs
}

When it happens

Trigger: Any failure inside the internal Nexus callback invocable (invocableInternal.Invoke) that delivers Nexus operation completion data to History for cross-shard callbacks — e.g. history shard unreachable, invalid callback data, or internal state errors — is logged and replaced by this error message.

Common situations: Developers see 'internal error, reference-id: <uuid>' in workflow events or Nexus operation results after a Nexus callback failed internally. Common causes: history service connectivity issues, cross-shard callback failures, or bugs in callback payload handling.

Related errors

• link type is empty
• link type contains invalid char (valid chars: alphanumeric,
• empty operation name
• empty operation token
• empty BaseURL

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/e98e4a752eedbd14. Report an issue: GitHub.