SigNoz/signoz · error · model.ApiError

error scanning fallback metadata: %v

Error message

error scanning fallback metadata: %v

What it means

Returned when rows.Scan fails while reading a row of fallback metrics metadata from the time_series_v4 query result. The result columns do not match the scan targets (metric name, type, description, temporality, is_monotonic, unit) — usually a schema drift issue where the table has different column types/order than expected.

Source

Thrown at pkg/query-service/app/clickhouseReader/reader.go:5240

			WHERE metric_name IN (%s)`, signozMetricDBName, signozTSTableNameV4, metricList)
		}
		valueCtx := context.WithValue(ctx, "clickhouse_max_threads", constants.MetricsExplorerClickhouseThreads)
		rows, err := r.db.Query(valueCtx, query)
		if err != nil {
			return cachedMetadata, &model.ApiError{Typ: "ClickhouseErr", Err: fmt.Errorf("error querying time_series_v4 to get metrics metadata: %v", err)}
		}
		defer rows.Close()
		for rows.Next() {
			metadata := new(model.UpdateMetricsMetadata)
			if err := rows.Scan(
				&metadata.MetricName,
				&metadata.MetricType,
				&metadata.Description,
				&metadata.Temporality,
				&metadata.IsMonotonic,
				&metadata.Unit,
			); err != nil {
				return cachedMetadata, &model.ApiError{Typ: "ClickhouseErr", Err: fmt.Errorf("error scanning fallback metadata: %v", err)}
			}

			cacheKey := constants.UpdatedMetricsMetadataCachePrefix + metadata.MetricName
			if cacheErr := r.cache.Set(ctx, orgID, cacheKey, metadata, 0); cacheErr != nil {
				r.logger.Error("Failed to cache fallback metadata", "metric_name", metadata.MetricName, errorsV2.Attr(cacheErr))
			}
			cachedMetadata[metadata.MetricName] = metadata
		}
		if rows.Err() != nil {
			return cachedMetadata, &model.ApiError{Typ: "ClickhouseErr", Err: fmt.Errorf("error scanning fallback metadata: %v", err)}
		}
	}
	return cachedMetadata, nil
}

func (r *ClickHouseReader) SearchTraces(ctx context.Context, params *model.SearchTracesParams) (*[]model.SearchSpansResult, error) {
	ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
		instrumentationtypes.TelemetrySignal:  telemetrytypes.SignalTraces.StringValue(),

View on GitHub (pinned to 5069bf80b0)

Solutions

1. Compare the actual column types via DESCRIBE TABLE signoz_metrics.time_series_v4 with the fields scanned (MetricName, MetricType, Description, Temporality, IsMonotonic, Unit)
2. Align query-service and ClickHouse schema versions by re-running SigNoz migrations so columns match
3. Wrap scanned columns with sql.Null* types or ensure the SELECT coalesces NULLs
4. Restart query-service after schema migration so prepared statements/statements cache are fresh

Example fix

// before
if err := rows.Scan(&metadata.MetricName, &metadata.MetricType, &metadata.Description, &metadata.Temporality, &metadata.IsMonotonic, &metadata.Unit); err != nil {
    return cachedMetadata, &model.ApiError{Typ: "ClickhouseErr", Err: fmt.Errorf("error scanning fallback metadata: %v", err)}
}

// after (null-safe scan)
var unit, desc sql.NullString
if err := rows.Scan(&metadata.MetricName, &metadata.MetricType, &desc, &metadata.Temporality, &metadata.IsMonotonic, &unit); err != nil { ... }
if unit.Valid { metadata.Unit = unit.String }
if desc.Valid { metadata.Description = desc.String }

// Verify column layout matches the scan before querying
DESCRIBE TABLE signoz_metrics.time_series_v4
// ensure temporality, is_monotonic, unit columns exist with expected types

// On ClickhouseErr from the fallback path, degrade gracefully to cached metadata only
if isClickhouseScanErr(err) {
    log.Warn("fallback metadata scan failed; serving cache only")
    return cachedMetadata, nil
}

When it happens

Trigger: GetMetricsMetadata fallback path scans a row whose columns mismatch the expected types: e.g. temporality or is_monotonic columns added/changed in a newer schema, NULL values in non-nullable scanned columns, or a ClickHouse version returning a type the driver cannot convert to the Go destination.

Common situations: SigNoz upgrade where time_series_v4 gained new columns but query-service is an older binary (or vice versa), partially applied migrations, or metrics without unit/description producing NULLs the scan cannot handle.

Related errors

• couldn't scan attrib value rows: %w
• error in reading data
• error while scanning metric name: %s
• error while scanning meter name: %s
• error while scanning rows: %s

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/b54ff6cae4c4e032. Report an issue: GitHub.