hashicorp/nomad · warning

failed to convert reattach config: %v

Error message

failed to convert reattach config: %v

What it means

During cleanupStalePlugins, each stored reattach config is converted to a go-plugin ReattachConfig via pstructs.ReattachConfigToGoPlugin. If conversion fails (malformed stored config), the error is appended to a multierror and cleanup continues with the next plugin. It indicates stale or corrupted reattach metadata persisted by a previous client run.

Source

Thrown at client/devicemanager/manager.go:289

func (m *manager) cleanupStalePlugins() error {

	// Read the old plugin state
	s, err := m.state.GetDevicePluginState()
	if err != nil {
		return fmt.Errorf("failed to read plugin state: %v", err)
	}

	// No state was stored so there is nothing to do.
	if s == nil {
		return nil
	}

	// For each plugin go through and try to shut it down
	var mErr multierror.Error
	for name, c := range s.ReattachConfigs {
		rc, err := pstructs.ReattachConfigToGoPlugin(c)
		if err != nil {
			_ = multierror.Append(&mErr, fmt.Errorf("failed to convert reattach config: %v", err))
			continue
		}

		instance, err := m.loader.Reattach(name, base.PluginTypeDevice, rc)
		if err != nil {
			_ = multierror.Append(&mErr, fmt.Errorf("failed to reattach to plugin %q: %v", name, err))
			continue
		}

		// Kill the instance
		instance.Kill()
	}

	return mErr.ErrorOrNil()
}

// storePluginReattachConfig is used as a callback to the instance managers and
// persists thhe plugin reattach configurations.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the stored device plugin state; delete stale reattach entries so the plugin is relaunched fresh.
2. Restart the nomad client — after cleanup the manager relaunches plugins and rewrites valid state.
3. Fix data_dir permissions/corruption if the state is repeatedly malformed.
4. Upgrade to a Nomad version that prunes stale reattach configs on shutdown.

Example fix

// before: stale reattach entry blocks startup cleanup
// state.ReattachConfigs: {"nvidia-gpu": {Pid: -1, ...}}
// after: clear stale plugin state and restart client
nomad client reload  # or remove the device-plugin state entry and restart nomad

// validate reattach config fields before use
func validReattach(c pstructs.ReattachConfig) bool { return c.Pid > 0 && c.Addr != nil && c.Protocol != "" }

func isConvertible(c interface{}) bool {
  _, err := pstructs.ReattachConfigToGoPlugin(c)
  return err == nil
}

if err := m.cleanupStalePlugins(); err != nil {
  // multierror may contain conversion/reattach issues; log and relaunch plugins fresh
  log.Warnf("stale plugin cleanup issues (non-fatal): %v", err)
}

When it happens

Trigger: The persisted ReattachConfigs map contains an entry with invalid/empty fields (bad PID, protocol, or address) that cannot be converted to a go-plugin reattach config.

Common situations: Client crashed mid-write leaving partial plugin state; state file hand-edited or from a different Nomad version with an incompatible reattach config schema.

Related errors

• failed to read plugin state: %v
• failed to reattach to plugin %q: %v
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6675a249f95862d5. Report an issue: GitHub.