SigNoz/signoz · error · model.InternalError

couldn't unmarshal metric labels json: %w

Error message

couldn't unmarshal metric labels json: %w

What it means

Internal error when the labels JSON string scanned from the time_series row cannot be unmarshalled into the label-map type. The stored last_received_labels value is malformed or not valid JSON.

Source

Thrown at pkg/query-service/app/clickhouseReader/reader.go:3430

	if rows.Next() {

		result = &model.MetricStatus{}
		var labelsJson string

		err := rows.Scan(
			&result.MetricName,
			&labelsJson,
			&result.LastReceivedTsMillis,
		)
		if err != nil {
			return nil, model.InternalError(fmt.Errorf(
				"couldn't scan metric status row: %w", err,
			))
		}

		err = json.Unmarshal([]byte(labelsJson), &result.LastReceivedLabels)
		if err != nil {
			return nil, model.InternalError(fmt.Errorf(
				"couldn't unmarshal metric labels json: %w", err,
			))
		}
	}

	return result, nil
}

func isColumn(tableStatement, attrType, field, datType string) bool {
	name := fmt.Sprintf("`%s`", utils.GetClickhouseColumnNameV2(attrType, datType, field))
	return strings.Contains(tableStatement, fmt.Sprintf("%s ", name))
}

func (r *ClickHouseReader) GetLogAggregateAttributes(ctx context.Context, req *v3.AggregateAttributeRequest) (*v3.AggregateAttributeResponse, error) {

	ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
		instrumentationtypes.TelemetrySignal:  telemetrytypes.SignalLogs.StringValue(),
		instrumentationtypes.CodeNamespace:    "clickhouse-reader",

View on GitHub (pinned to 5069bf80b0)

Solutions

  1. Select last_received_labels for the affected metric manually and inspect the value
  2. If data is corrupt, delete/re-ingest the affected series or guard unmarshal per-row
  3. Verify query-service and schema versions match
  4. Add a fallback that tolerates empty/invalid label JSON

Example fix

// before
err = json.Unmarshal([]byte(labelsJson), &result.LastReceivedLabels)
if err != nil { return nil, model.InternalError(...) }

// after
if labelsJson != "" {
    if err := json.Unmarshal([]byte(labelsJson), &result.LastReceivedLabels); err != nil {
        r.logger.Error("bad labels json", errorsV2.Attr(err))
    }
}
Defensive patterns

Strategy: try-catch

Validate before calling

if !json.Valid([]byte(labelsJson)) {
    log.Warn("invalid labels json; ignoring")
    continue
}

Type guard

func isValidLabelsJSON(s string) bool { return s == "" || json.Valid([]byte(s)) }

Try / catch

if err := json.Unmarshal(data, &v); err != nil {
    log.Warn("bad labels json", err); v = nil // degrade, don't fail request
}

Prevention

When it happens

Trigger: Corrupted label JSON in ClickHouse (truncated write, wrong encoding), or a schema change where the column contains a non-JSON value (e.g. empty string or array format).

Common situations: Data corruption from interrupted ingester writes; older SigNoz versions storing labels differently; manual edits to the table.

Related errors

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/768792a9461857a5. Report an issue: GitHub.