{"record":{"id":"6aac1b0ea858f3cf","repo":"hashicorp/nomad","slug":"singleton-plugin-exited","errorCode":null,"errorMessage":"singleton plugin exited","messagePattern":"singleton plugin exited","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"helper/pluginutils/singleton/singleton.go","lineNumber":20,"sourceCode":"// SPDX-License-Identifier: BUSL-1.1\n\npackage singleton\n\nimport (\n\t\"fmt\"\n\t\"sync\"\n\n\tlog \"github.com/hashicorp/go-hclog\"\n\tplugin \"github.com/hashicorp/go-plugin\"\n\t\"github.com/hashicorp/nomad/helper/pluginutils/loader\"\n\t\"github.com/hashicorp/nomad/plugins/base\"\n)\n\nvar (\n\t// SingletonPluginExited is returned when the dispense is called and the\n\t// existing plugin has exited. The caller should retry, and this will issue\n\t// a new plugin instance.\n\tSingletonPluginExited = fmt.Errorf(\"singleton plugin exited\")\n)\n\n// SingletonLoader is used to only load a single external plugin at a time.\ntype SingletonLoader struct {\n\t// Loader is the underlying plugin loader that we wrap to give a singleton\n\t// behavior.\n\tloader loader.PluginCatalog\n\n\t// instances is a mapping of the plugin to a future which holds a plugin\n\t// instance\n\tinstances    map[loader.PluginID]*future\n\tinstanceLock sync.Mutex\n\n\t// logger is the logger used by the singleton\n\tlogger log.Logger\n}\n\n// NewSingletonLoader wraps a plugin catalog and provides singleton behavior on","sourceCodeStart":2,"sourceCodeEnd":38,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/pluginutils/singleton/singleton.go#L2-L38","documentation":"SingletonPluginExited is a sentinel error returned by the SingletonLoader's dispense/getPlugin when the currently loaded singleton plugin process has exited. It is not fatal: callers are expected to compare with it and retry dispense, which launches a new plugin instance.","triggerScenarios":"Dispense through a SingletonLoader when the cached plugin process died; seen in client/devicemanager/instance.go:284 and client/pluginmanager/drivermanager/instance.go:213.","commonSituations":"Driver/device plugin binary crashing (OOM, segfault), plugin upgraded on disk while running, or agent-side kill of the plugin process.","solutions":["Retry the dispense (the library contract) — callers like device/driver managers already do this automatically","Check why the plugin exited: inspect its stderr/stdout logs for panics or OOM kills","Upgrade or fix the plugin binary that keeps dying","Ensure sufficient memory/resources so the plugin process is not killed"],"exampleFix":"// before\npluginInstance, err = i.loader.Dispense(...)\nif err != nil { return err }\n// after\nif err == singleton.SingletonPluginExited {\n    pluginInstance, err = i.loader.Dispense(...) // retry spawns new instance\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"inst, err := i.loader.Dispense(...)\nif err != nil {\n    if errors.Is(err, singleton.SingletonPluginExited) {\n        inst, err = i.loader.Dispense(...) // one retry launches a new instance\n    }\n    if err != nil { return err }\n}","preventionTips":["Monitor plugin process health and restart counts","Always compare with errors.Is/== against the sentinel and retry once","Capture plugin stdout/stderr to diagnose why it exited"],"tags":["plugin-loader","singleton","retry"],"backgroundTag":"plugin-process-exited","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}