hashicorp/nomad · error

error parsing: root should be an object

Error message

error parsing: root should be an object

What it means

Nomad's controllerUnpublishVolume (csi_endpoint.go) failed to look up the CSI plugin record for the volume in the state store snapshot via snap.CSIPluginByID. The wrapped error indicates an internal state-store query failure rather than a missing plugin. This is a server-side error during volume unpublish reconciliation.

Source

Thrown at acl/policy.go:760

	}

	// Manually parse the policy to fix blocks without labels.
	//
	// Due to a bug in the way HCL decodes files, a block without a label may
	// return an incorrect key value and make it impossible to determine if the
	// key was set by the user or incorrectly set by the decoder.
	//
	// By manually parsing the file we are able to determine if the label is
	// missing in the file and set them to an empty string so the policy
	// validation can return the appropriate errors.
	root, err := hcl.Parse(rules)
	if err != nil {
		return fmt.Errorf("failed to parse policy: %w", err)
	}

	list, ok := root.Node.(*ast.ObjectList)
	if !ok {
		return errors.New("error parsing: root should be an object")
	}

	nsList := list.Filter("namespace")
	for i, nsObj := range nsList.Items {
		// Fix missing namespace key.
		if len(nsObj.Keys) == 0 {
			p.Namespaces[i].Name = ""
		}
		if i > 0 {
			p.removeExtraKey("namespace")
		}

		// Fix missing variable paths.
		nsOT, ok := nsObj.Val.(*ast.ObjectType)
		if !ok {
			continue
		}
		varsList := nsOT.List.Filter("variables")

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped inner error (%v) in the Nomad server logs to identify the state-store failure
2. Retry the unpublish; the claim runner re-invokes controllerUnpublishVolume with backoff
3. Check server health and raft stability; wait out leader elections before further retries
4. If persistent, restart the Nomad server and verify state store integrity with nomad operator raft logs

Example fix

// caller-side: tolerate transient state-store failures during unpublish
// before
err := v.controllerUnpublishVolume(vol, claim, cs)
if err != nil { return err }
// after
err := v.controllerUnpublishVolume(vol, claim, cs)
if err != nil && strings.Contains(err.Error(), "could not query plugin") {
    v.logger.Warn("transient plugin lookup failure; will retry", "vol", vol.ID)
    return nil // claim requeued by checkpoint loop
}
return err

// verify plugin resolvable before triggering unpublish tooling
p, err := api.Plugins().Get(pluginID, nil)
if err != nil || p == nil { /* re-register plugin first */ }

if err := unpublish(vol); err != nil && strings.Contains(err.Error(), "could not query plugin") {
    // transient state-store failure: backoff and retry
    time.Sleep(backoff); retry()
}

When it happens

Trigger: A volume claim (Unpublish RPC) targets a volume whose PluginID causes the state-store query to return a non-nil error, e.g. during a blocking query/watch error or corrupted snapshot iteration while the server reconciles the claim.

Common situations: State store issues on a heavily loaded leader, raft snapshot inconsistencies, or querying during leader elections/failover while unpublish is being retried by the claim runner.

Related errors

• plugin lookup error: %s %v
• csi_plugin lookup failed: %s %v
• cannot specify Accessor ID
• failed to read dynamic plugin registry state: %v
• error getting plugin: %s, %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/53770d8a9218e2a1. Report an issue: GitHub.