hashicorp/nomad · error

panic(err)

Error message

panic(err)

What it means

When the executor's stats call returns a recoverable error, the gRPC server attaches a RecoverableError detail to a FailedPrecondition status. Building that status should never fail; if st.WithDetails errors, the code panics because the failure indicates a broken proto registration that would persist on every retry.

Source

Thrown at drivers/shared/executor/grpc_server.go:120

	return &proto.VersionResponse{
		Version: v.Version,
	}, nil
}

func (s *grpcExecutorServer) Stats(req *proto.StatsRequest, stream proto.Executor_StatsServer) error {
	interval := time.Duration(req.Interval)
	if interval == 0 {
		interval = time.Second
	}

	outCh, err := s.impl.Stats(stream.Context(), interval)
	if err != nil {
		if rec, ok := err.(structs.Recoverable); ok {
			st := status.New(codes.FailedPrecondition, rec.Error())
			st, err := st.WithDetails(&sproto.RecoverableError{Recoverable: rec.IsRecoverable()})
			if err != nil {
				// If this error, it will always error
				panic(err)
			}
			return st.Err()
		}
		return err
	}

	for resp := range outCh {
		pbStats, err := drivers.TaskStatsToProto(resp)
		if err != nil {
			return err
		}

		presp := &proto.StatsResponse{
			Stats: pbStats,
		}

		// Send the stats
		if err := stream.Send(presp); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure the drivers/shared/execsproto package is imported and linked into the build so proto details can marshal
2. Rebuild the Nomad binary from an unmodified source tree
3. In forks/tests, make sure custom recoverable errors are serializable into the sproto.RecoverableError detail

defer func() {
    if r := recover(); r != nil {
        logger.Error("grpc status detail build failed", "panic", r)
        resp = status.Error(codes.FailedPrecondition, "recoverable")
    }
}()

When it happens

Trigger: Returning a structs.Recoverable error from the executor whose status-with-details marshaling fails — practically only when the RecoverableError proto is not registered/linked in the binary.

Common situations: Mispackaged or partially built Nomad binary missing the sproto registration; vendoring/build changes that drop the proto import; injecting a custom Recoverable error type into the executor in tests or forks.

Related errors

• nil response from plugin.NodeExpandVolume
• ErrCgroupMustBeSet
• user name must contain domain
• invalid acl policy: %v
• CSI plugin failed to register: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/dab5e3c0f4016588. Report an issue: GitHub.