prometheus/node_exporter · critical

panic(err)

Error message

panic(err)

What it means

kstatToFloat panics when ks.GetNamed(kstatKey) fails on the Solaris/illumos kstat "unix:0:system_misc" named statistic. GetNamed returns an error if the named kstat (e.g. avenrun_1min) does not exist or the kstat data could not be read; this collector turns that into an unrecoverable panic rather than an error, crashing the exporter's scrape goroutine.

Solutions

1. Verify the kstat exists on the affected host: run `kstat -p unix:0:system_misc:avenrun_1min` and confirm output.
2. Run node_exporter in the global zone (or grant the zone access to system_misc kstats) so avenrun_* statistics are visible.
3. Patch kstatToFloat to return (float64, error) and propagate the error to getLoad instead of panicking.
4. Build with the -tags noloadavg build tag to exclude the loadavg collector if load average cannot be supported on the target platform.

Example fix

// before
func kstatToFloat(ks *kstat.KStat, kstatKey string) float64 {
	kstatValue, err := ks.GetNamed(kstatKey)
	if err != nil {
		panic(err)
	}

// after
func kstatToFloat(ks *kstat.KStat, kstatKey string) (float64, error) {
	kstatValue, err := ks.GetNamed(kstatKey)
	if err != nil {
		return 0, fmt.Errorf("couldn't get kstat %s: %w", kstatKey, err)
	}

// before scraping loadavg on Solaris/illumos
out, err := exec.Command("kstat", "-p", "unix:0:system_misc:avenrun_1min").Output()
if err != nil || len(out) == 0 {
	log.Println("avenrun_1min kstat not available; loadavg metrics disabled")
}

func hasAvenrunKstat(ks *kstat.KStat) bool {
	_, err := ks.GetNamed("avenrun_1min")
	return err == nil
}

// Go has no catch for panic; recover at the scrape boundary
func safeGetLoad() (v []float64, err error) {
	defer func() {
		if r := recover(); r != nil {
			err = fmt.Errorf("loadavg kstat panic: %v", r)
		}
	}()
	return getLoad()
}

When it happens

Trigger: Calling getLoad -> kstatToFloat(ks, "avenrun_1min"/"avenrun_5min"/"avenrun_15min") on Solaris/illumos when the system_misc kstat lacks the requested avenrun_* named statistic or GetNamed cannot read the kstat value (kstat chain changed, snapshot failed, permission problem opening kstats).

Common situations: Running node_exporter in a Solaris zone/branded zone where system_misc avenrun kstats are not exposed; illumos distribution or kernel revision that renamed or dropped the avenrun_* named stats; kstat permissions restricting access to the unix:0:system_misc instance.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• unknown metric type

AI-assisted analysis of prometheus/node_exporter@17ddd77c59 (2026-09-07). Data as JSON: /api/errors/684acd18efd1af97. Report an issue: GitHub.

Appendix: source

Thrown at collector/loadavg_solaris.go:32

//go:build !noloadavg

package collector

import (
	"fmt"
	"strconv"

	"github.com/illumos/go-kstat"
)

// #include <sys/param.h>
import "C"

func kstatToFloat(ks *kstat.KStat, kstatKey string) float64 {
	kstatValue, err := ks.GetNamed(kstatKey)

	if err != nil {
		panic(err)
	}

	kstatLoadavg, err := strconv.ParseFloat(
		fmt.Sprintf("%.2f", float64(kstatValue.UintVal)/C.FSCALE), 64)

	if err != nil {
		panic(err)
	}

	return kstatLoadavg
}

func getLoad() ([]float64, error) {
	tok, err := kstat.Open()
	if err != nil {
		panic(err)
	}

View on GitHub (pinned to 17ddd77c59)