What causes "failed to unmarshal %s: %v" in istio/istio?

Check the wrapped json error in the log — it gives the exact offset/field that failed

failed to unmarshal %s: %v

Error message

failed to unmarshal %s: %v

What it means

NewServer failed to json.Unmarshal the PrometheusScrapingConfig setting (looked up via PrometheusScrapingConfig.Lookup()) into the PrometheusScrapeConfiguration struct. This setting controls application metrics merging on port 15020 and is only parsed for SidecarProxy nodes. The value is a JSON string; any syntax error or wrong field type aborts agent startup.

Source

Thrown at pilot/cmd/pilot-agent/status/server.go:275

		registry:              registry,
		shutdown:              config.Shutdown,
		drain:                 config.TriggerDrain,
		disableDrain:          config.DisableDrain,
		maxAppBodyBytes:       defaultMaxAppMetricsBodyBytes,
	}
	if LegacyLocalhostProbeDestination.Get() {
		s.appProbersDestination = "localhost"
	}

	// Enable prometheus server if its configured and a sidecar
	// Because port 15020 is exposed in the gateway Services, we cannot safely serve this endpoint
	// If we need to do this in the future, we should use envoy to do routing or have another port to make this internal
	// only. For now, its not needed for gateway, as we can just get Envoy stats directly, but if we
	// want to expose istio-agent metrics we may want to revisit this.
	if cfg, f := PrometheusScrapingConfig.Lookup(); config.NodeType == model.SidecarProxy && f {
		var prom PrometheusScrapeConfiguration
		if err := json.Unmarshal([]byte(cfg), &prom); err != nil {
			return nil, fmt.Errorf("failed to unmarshal %s: %v", PrometheusScrapingConfig.Name, err)
		}
		log.Infof("Prometheus scraping configuration: %v", prom)
		if prom.Scrape != "false" {
			s.prometheus = &prom
			if len(s.prometheus.Targets) == 0 {
				// Legacy path: apply defaults and synthesize a single Targets entry.
				if s.prometheus.Path == "" {
					s.prometheus.Path = "/metrics"
				}
				if s.prometheus.Port == "" {
					s.prometheus.Port = "80"
				}
				if s.prometheus.Port == strconv.Itoa(int(config.StatusPort)) {
					return nil, fmt.Errorf("invalid prometheus scrape configuration: "+
						"application port is the same as agent port, which may lead to a recursive loop. "+
						"Ensure pod does not have prometheus.io/port=%d label, or that injection is not happening multiple times", config.StatusPort)
				}
				s.prometheus.Targets = []ScrapeTarget{{Port: s.prometheus.Port, Path: s.prometheus.Path}}

View on GitHub (pinned to 8dc789c5cf)

Solutions

Check the wrapped json error in the log — it gives the exact offset/field that failed
Validate the value with a JSON linter and against the PrometheusScrapeConfiguration shape: scrape/path/port are strings, targets is an array of {port, path} strings
Fix quoting: in YAML annotations use the '>' or '|' block style or escape double quotes so the delivered value is one JSON document
If you do not need app-metrics merging, set scrape to "false" (still valid JSON) or remove the config entirely

Example fix

# before
annotations:
  prometheus.istio.io/merge-metrics-config: '{ scrape: true, port: 8080, }'  # YAML-ish, trailing comma
# after
annotations:
  prometheus.istio.io/merge-metrics-config: |
    {"scrape": "true", "targets": [{"port": "8080", "path": "/metrics"}]}

Defensive patterns

Strategy: validation

Validate before calling

// Validate the scrape config JSON exactly like the agent before injecting/starting
var prom PrometheusScrapeConfiguration
if err := json.Unmarshal([]byte(cfgString), &prom); err != nil {
    return fmt.Errorf("bad PrometheusScrapingConfig, fix before rollout: %w", err)
}

Prevention

Run every hand-edited scrape config through a JSON schema/linter in CI
Write JSON in annotations with YAML block scalars (|) to avoid quote mangling
Remember ports and paths are strings in this struct, not numbers

When it happens

Trigger: PrometheusScrapingConfig is set (annotation/mesh/proxyConfig surface) and node type is SidecarProxy, but the string is not valid JSON for {scrape, path, port, targets} — e.g. trailing commas, single quotes, YAML instead of JSON, or port given as a number instead of a string.

Common situations: Hand-writing the scrape config JSON in an annotation or meshConfig and quoting it wrong; passing YAML because the surrounding config file is YAML; version upgrade that changed the expected schema (e.g. introduction of the targets field); copy-paste from docs that mangled quotes.

Related errors

AI-assisted analysis of istio/istio@8dc789c5cf (2026-08-15). Data as JSON: /api/errors/68c6d29f1f3cbbba. Report an issue: GitHub.