SigNoz/signoz · error · model.ApiError

error in processing ttl_status check sql query

Error message

error in processing ttl_status check sql query

What it means

ApiError (ErrorExec) from setTTLLogs when checkTTLStatusItem fails for any table in the TTL set — the query that reads the current ttl_status (pending/running state) errors out, so a new TTL change cannot be gated safely. Distinct from the adjacent 'TTL is already running' (ErrorConflict) which fires when status is legitimately pending.

Solutions

1. Inspect logs for why checkTTLStatusItem failed
2. Verify the ttl_status table exists (run migrations) and is readable by the CH user
3. Fix ClickHouse connectivity/permissions and retry the TTL update

if !tableExists(ctx, ch, metaDB, ttlStatusTable) {
    return fmt.Errorf("ttl_status table missing; run migrations")
}

func isTtlStatusErr(apiErr *model.ApiError) bool {
    return apiErr != nil && apiErr.Typ == model.ErrorExec && strings.Contains(apiErr.Err.Error(), "ttl_status check sql query")
}

_, apiErr := reader.SetTTL(ctx, orgID, params)
if isTtlStatusErr(apiErr) {
    logMigrationsNeeded()
    return apiErr // fix schema, don't retry
} else if apiErr != nil && strings.Contains(apiErr.Err.Error(), "already running") {
    // wait for pending TTL, then re-queue
}

When it happens

Trigger: Initiating a logs TTL change while the ttl_status lookup query fails: missing/corrupt ttl_status table, CH connection errors, or permission problems on the metadata table.

Common situations: Deployments where TTL metadata migrations didn't run, CH user lacking access to the TTL status table, or transient CH failures during retention updates.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• error in processing TTL
• error while getting ttl. Err=
• error fetching metric metadata
• error getting updated metrics metadata
• internal

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/47c161cb26c8f062. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query-service/app/clickhouseReader/reader.go:955

	uuid := strings.Replace(uuidWithHyphen.String(), "-", "", -1)

	coldStorageDuration := -1
	if len(params.ColdStorageVolume) > 0 {
		coldStorageDuration = int(params.ToColdStorageDuration)
	}

	tableNameArray := []string{
		r.logsDB + "." + r.logsLocalTableV2,
		r.logsDB + "." + r.logsResourceLocalTableV2,
		getLocalTableName(r.logsDB + "." + r.logsAttributeKeys),
		getLocalTableName(r.logsDB + "." + r.logsResourceKeys),
	}

	// check if there is existing things to be done
	for _, tableName := range tableNameArray {
		statusItem, apiErr := r.checkTTLStatusItem(ctx, orgID, tableName)
		if apiErr != nil {
			return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error in processing ttl_status check sql query")}
		}
		if statusItem.Status == retentiontypes.TTLSettingStatusPending {
			return nil, &model.ApiError{Typ: model.ErrorConflict, Err: fmt.Errorf("TTL is already running")}
		}
	}

	// TTL query for logs_v2 table
	ttlLogsV2 := fmt.Sprintf(
		"ALTER TABLE %v ON CLUSTER %s MODIFY TTL toDateTime(timestamp / 1000000000) + "+
			"INTERVAL %v SECOND DELETE", tableNameArray[0], r.cluster, params.DelDuration)
	if len(params.ColdStorageVolume) > 0 {
		ttlLogsV2 += fmt.Sprintf(", toDateTime(timestamp / 1000000000)"+
			" + INTERVAL %v SECOND TO VOLUME '%s'",
			params.ToColdStorageDuration, params.ColdStorageVolume)
	}

	// TTL query for logs_v2_resource table
	// adding 1800 as our bucket size is 1800 seconds

View on GitHub (pinned to 5069bf80b0)