SigNoz/signoz · error · model.ApiError

error in processing ttl_status check sql query

Error message

error in processing ttl_status check sql query

What it means

ApiError (ErrorExec) from setTTLLogs when checkTTLStatusItem fails for any table in the TTL set — the query that reads the current ttl_status (pending/running state) errors out, so a new TTL change cannot be gated safely. Distinct from the adjacent 'TTL is already running' (ErrorConflict) which fires when status is legitimately pending.

Source

Thrown at pkg/query-service/app/clickhouseReader/reader.go:955

	uuid := strings.Replace(uuidWithHyphen.String(), "-", "", -1)

	coldStorageDuration := -1
	if len(params.ColdStorageVolume) > 0 {
		coldStorageDuration = int(params.ToColdStorageDuration)
	}

	tableNameArray := []string{
		r.logsDB + "." + r.logsLocalTableV2,
		r.logsDB + "." + r.logsResourceLocalTableV2,
		getLocalTableName(r.logsDB + "." + r.logsAttributeKeys),
		getLocalTableName(r.logsDB + "." + r.logsResourceKeys),
	}

	// check if there is existing things to be done
	for _, tableName := range tableNameArray {
		statusItem, apiErr := r.checkTTLStatusItem(ctx, orgID, tableName)
		if apiErr != nil {
			return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error in processing ttl_status check sql query")}
		}
		if statusItem.Status == retentiontypes.TTLSettingStatusPending {
			return nil, &model.ApiError{Typ: model.ErrorConflict, Err: fmt.Errorf("TTL is already running")}
		}
	}

	// TTL query for logs_v2 table
	ttlLogsV2 := fmt.Sprintf(
		"ALTER TABLE %v ON CLUSTER %s MODIFY TTL toDateTime(timestamp / 1000000000) + "+
			"INTERVAL %v SECOND DELETE", tableNameArray[0], r.cluster, params.DelDuration)
	if len(params.ColdStorageVolume) > 0 {
		ttlLogsV2 += fmt.Sprintf(", toDateTime(timestamp / 1000000000)"+
			" + INTERVAL %v SECOND TO VOLUME '%s'",
			params.ToColdStorageDuration, params.ColdStorageVolume)
	}

	// TTL query for logs_v2_resource table
	// adding 1800 as our bucket size is 1800 seconds

View on GitHub (pinned to 5069bf80b0)

Solutions

1. Inspect logs for why checkTTLStatusItem failed
2. Verify the ttl_status table exists (run migrations) and is readable by the CH user
3. Fix ClickHouse connectivity/permissions and retry the TTL update

if !tableExists(ctx, ch, metaDB, ttlStatusTable) {
    return fmt.Errorf("ttl_status table missing; run migrations")
}

func isTtlStatusErr(apiErr *model.ApiError) bool {
    return apiErr != nil && apiErr.Typ == model.ErrorExec && strings.Contains(apiErr.Err.Error(), "ttl_status check sql query")
}

_, apiErr := reader.SetTTL(ctx, orgID, params)
if isTtlStatusErr(apiErr) {
    logMigrationsNeeded()
    return apiErr // fix schema, don't retry
} else if apiErr != nil && strings.Contains(apiErr.Err.Error(), "already running") {
    // wait for pending TTL, then re-queue
}

When it happens

Trigger: Initiating a logs TTL change while the ttl_status lookup query fails: missing/corrupt ttl_status table, CH connection errors, or permission problems on the metadata table.

Common situations: Deployments where TTL metadata migrations didn't run, CH user lacking access to the TTL status table, or transient CH failures during retention updates.

Related errors

• error in processing TTL
• error while getting ttl. Err=%v
• SetTTLV2 only supported
• error getting updated metrics metadata: %s
• error fetching metric metadata: %s

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/47c161cb26c8f062. Report an issue: GitHub.