{"record":{"id":"e98e4a752eedbd14","repo":"temporalio/temporal","slug":"internal-error-reference-id-v","errorCode":null,"errorMessage":"internal error, reference-id: %v","messagePattern":"internal error, reference-id: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"chasm/lib/callback/invocable_internal.go","lineNumber":32,"sourceCode":"\tcallbackspb \"go.temporal.io/server/chasm/lib/callback/gen/callbackpb/v1\"\n\t\"go.temporal.io/server/common/log\"\n\t\"go.temporal.io/server/common/log/tag\"\n\t\"go.temporal.io/server/common/namespace\"\n\tcommonnexus \"go.temporal.io/server/common/nexus\"\n\t\"go.temporal.io/server/common/nexus/nexusrpc\"\n\t\"google.golang.org/grpc/codes\"\n\t\"google.golang.org/grpc/status\"\n\t\"google.golang.org/protobuf/proto\"\n\t\"google.golang.org/protobuf/types/known/timestamppb\"\n)\n\n// logInternalError emits a log statement for internalMsg, tagged with both\n// internalErr and a reference-id. An opaque error containing the reference-id is\n// returned. Intended to be used to hide internal errors from end users.\nfunc logInternalError(logger log.Logger, internalMsg string, internalErr error) error {\n\treferenceID := uuid.NewString()\n\tlogger.Error(internalMsg, tag.Error(internalErr), tag.String(\"reference-id\", referenceID))\n\treturn fmt.Errorf(\"internal error, reference-id: %v\", referenceID)\n}\n\n// invocableInternal is an invocable that delivers the Nexus operation completion data to History for cross-shard\n// callbacks.\ntype invocableInternal struct {\n\tcallback   *callbackspb.Callback_Nexus\n\tattempt    int32\n\tcompletion nexusrpc.CompleteOperationOptions\n\trequestID  string\n}\n\nfunc (c invocableInternal) WrapError(result invocationResult, err error) error {\n\t// Return the invocation result error if present\n\tif resultErr := result.error(); resultErr != nil {\n\t\treturn resultErr\n\t}\n\n\treturn err","sourceCodeStart":14,"sourceCodeEnd":50,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/chasm/lib/callback/invocable_internal.go#L14-L50","documentation":"logInternalError logs an internal error with full detail and returns a sanitized opaque error containing only a fresh UUID reference-id. The library throws this to hide internal (potentially sensitive) Nexus callback errors from end users while still allowing operators to correlate the user-visible message with the detailed server log entry.","triggerScenarios":"Any failure inside the internal Nexus callback invocable (invocableInternal.Invoke) that delivers Nexus operation completion data to History for cross-shard callbacks — e.g. history shard unreachable, invalid callback data, or internal state errors — is logged and replaced by this error message.","commonSituations":"Developers see 'internal error, reference-id: <uuid>' in workflow events or Nexus operation results after a Nexus callback failed internally. Common causes: history service connectivity issues, cross-shard callback failures, or bugs in callback payload handling.","solutions":["Copy the reference-id and search the history/worker service logs for it to find the underlying internalErr with full detail","Verify history service is healthy and reachable from the service where the Nexus callback was invoked","Check Nexus operation callback data integrity; re-trigger the callback if a transient history error occurred","If reproducible, file a bug with the reference-id and logs — most internal errors indicate a real defect"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go\nerr := invocable.Invoke(ctx)\nif err != nil && strings.HasPrefix(err.Error(), \"internal error, reference-id:\") {\n    refID := strings.TrimPrefix(err.Error(), \"internal error, reference-id: \")\n    logger.Warn(\"nexus callback failed internally\", \"reference-id\", refID)\n    // surface refID to user; look it up in server logs\n}","preventionTips":["Monitor service logs for reference-id-tagged internal errors and alert on spikes","Keep history service reachable and healthy — most callback internal errors are cross-shard delivery failures","Add the returned reference-id to user-facing incident reports for log correlation","Never retry blindly; first retrieve the underlying cause via the reference-id"],"tags":["nexus","callback","internal-error","logging"],"backgroundTag":"internal-error-reference-id","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}