SigNoz/signoz · error · model.ApiError

error in processing TTL

Error message

error in processing TTL

What it means

ApiError (ErrorExec) from setTTLLogs when the hasCustomRetentionColumn helper itself errors while probing the ClickHouse schema. Distinguishing v1 vs v2 retention failed, so the TTL operation cannot proceed safely.

Source

Thrown at pkg/query-service/app/clickhouseReader/reader.go:933

func getLocalTableName(tableName string) string {

	tableNameSplit := strings.Split(tableName, ".")
	return tableNameSplit[0] + "." + strings.Split(tableNameSplit[1], "distributed_")[1]

}

func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, params *retentiontypes.TTLParams) (*retentiontypes.SetTTLResponseItem, *model.ApiError) {
	ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
		instrumentationtypes.TelemetrySignal:  telemetrytypes.SignalLogs.StringValue(),
		instrumentationtypes.CodeNamespace:    "clickhouse-reader",
		instrumentationtypes.CodeFunctionName: "setTTLLogs",
	})
	hasCustomRetention, err := r.hasCustomRetentionColumn(ctx)
	if hasCustomRetention {
		return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("SetTTLV2 only supported")}
	}
	if err != nil {
		return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error in processing TTL")}
	}
	// uuid is used as transaction id
	uuidWithHyphen := uuid.New()
	uuid := strings.Replace(uuidWithHyphen.String(), "-", "", -1)

	coldStorageDuration := -1
	if len(params.ColdStorageVolume) > 0 {
		coldStorageDuration = int(params.ToColdStorageDuration)
	}

	tableNameArray := []string{
		r.logsDB + "." + r.logsLocalTableV2,
		r.logsDB + "." + r.logsResourceLocalTableV2,
		getLocalTableName(r.logsDB + "." + r.logsAttributeKeys),
		getLocalTableName(r.logsDB + "." + r.logsResourceKeys),
	}

	// check if there is existing things to be done

View on GitHub (pinned to 5069bf80b0)

Solutions

1. Check logs for the underlying probe failure
2. Verify the CH user can read system.columns / describe the logs DB
3. Retry when ClickHouse is reachable
4. If permissions are the issue, grant metadata read access to the query-service CH user

// Grant and verify system.columns read before TTL ops
if !canReadSystemColumns(ctx, ch) { return fmt.Errorf("CH user lacks metadata grants") }

func isTtlProbeErr(apiErr *model.ApiError) bool {
    return apiErr != nil && apiErr.Typ == model.ErrorExec && strings.Contains(apiErr.Err.Error(), "error in processing TTL")
}

apiErr := reader.SetTTL(ctx, orgID, params)
if isTtlProbeErr(apiErr) {
    time.Sleep(3*time.Second)
    apiErr = reader.SetTTL(ctx, orgID, params) // transient probe failure
}

When it happens

Trigger: Calling the logs TTL API while the schema-introspection query (checking for the custom_retention column) fails — CH unreachable, permissions denied on system.columns, or a malformed/failed DESCRIBE during the probe.

Common situations: Transient ClickHouse outages during TTL calls, restricted CH user lacking system-table read grants, or interrupted connections mid-probe.

Related errors

• error in processing ttl_status check sql query
• error while getting ttl. Err=%v
• SetTTLV2 only supported
• type and duration param cannot be empty from the query
• type param should be metrics|traces|logs, got %v

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/bace5d5c346d4b48. Report an issue: GitHub.