{"record":{"id":"bace5d5c346d4b48","repo":"SigNoz/signoz","slug":"error-in-processing-ttl","errorCode":null,"errorMessage":"error in processing TTL","messagePattern":"error in processing TTL","errorType":"exception","errorClass":"model.ApiError","httpStatus":500,"severity":"error","filePath":"pkg/query-service/app/clickhouseReader/reader.go","lineNumber":933,"sourceCode":"func getLocalTableName(tableName string) string {\n\n\ttableNameSplit := strings.Split(tableName, \".\")\n\treturn tableNameSplit[0] + \".\" + strings.Split(tableNameSplit[1], \"distributed_\")[1]\n\n}\n\nfunc (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, params *retentiontypes.TTLParams) (*retentiontypes.SetTTLResponseItem, *model.ApiError) {\n\tctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{\n\t\tinstrumentationtypes.TelemetrySignal:  telemetrytypes.SignalLogs.StringValue(),\n\t\tinstrumentationtypes.CodeNamespace:    \"clickhouse-reader\",\n\t\tinstrumentationtypes.CodeFunctionName: \"setTTLLogs\",\n\t})\n\thasCustomRetention, err := r.hasCustomRetentionColumn(ctx)\n\tif hasCustomRetention {\n\t\treturn nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf(\"SetTTLV2 only supported\")}\n\t}\n\tif err != nil {\n\t\treturn nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf(\"error in processing TTL\")}\n\t}\n\t// uuid is used as transaction id\n\tuuidWithHyphen := uuid.New()\n\tuuid := strings.Replace(uuidWithHyphen.String(), \"-\", \"\", -1)\n\n\tcoldStorageDuration := -1\n\tif len(params.ColdStorageVolume) > 0 {\n\t\tcoldStorageDuration = int(params.ToColdStorageDuration)\n\t}\n\n\ttableNameArray := []string{\n\t\tr.logsDB + \".\" + r.logsLocalTableV2,\n\t\tr.logsDB + \".\" + r.logsResourceLocalTableV2,\n\t\tgetLocalTableName(r.logsDB + \".\" + r.logsAttributeKeys),\n\t\tgetLocalTableName(r.logsDB + \".\" + r.logsResourceKeys),\n\t}\n\n\t// check if there is existing things to be done","sourceCodeStart":915,"sourceCodeEnd":951,"githubUrl":"https://github.com/SigNoz/signoz/blob/5069bf80b08f1f00d7e014eccc09902f9871004f/pkg/query-service/app/clickhouseReader/reader.go#L915-L951","documentation":"ApiError (ErrorExec) from setTTLLogs when the hasCustomRetentionColumn helper itself errors while probing the ClickHouse schema. Distinguishing v1 vs v2 retention failed, so the TTL operation cannot proceed safely.","triggerScenarios":"Calling the logs TTL API while the schema-introspection query (checking for the custom_retention column) fails — CH unreachable, permissions denied on system.columns, or a malformed/failed DESCRIBE during the probe.","commonSituations":"Transient ClickHouse outages during TTL calls, restricted CH user lacking system-table read grants, or interrupted connections mid-probe.","solutions":["Check logs for the underlying probe failure","Verify the CH user can read system.columns / describe the logs DB","Retry when ClickHouse is reachable","If permissions are the issue, grant metadata read access to the query-service CH user"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Grant and verify system.columns read before TTL ops\nif !canReadSystemColumns(ctx, ch) { return fmt.Errorf(\"CH user lacks metadata grants\") }","typeGuard":"func isTtlProbeErr(apiErr *model.ApiError) bool {\n    return apiErr != nil && apiErr.Typ == model.ErrorExec && strings.Contains(apiErr.Err.Error(), \"error in processing TTL\")\n}","tryCatchPattern":"apiErr := reader.SetTTL(ctx, orgID, params)\nif isTtlProbeErr(apiErr) {\n    time.Sleep(3*time.Second)\n    apiErr = reader.SetTTL(ctx, orgID, params) // transient probe failure\n}","preventionTips":["Give the query-service CH user metadata read grants","Alert on TTL probe failures as CH health signals","Retry TTL changes idempotently after CH recovery"],"tags":["ttl","retention","clickhouse","permissions"],"backgroundTag":"database-query-failed","analyzedSha":"5069bf80b08f1f00d7e014eccc09902f9871004f","analyzedAt":"2026-08-28T06:22:12.824Z","schemaVersion":2},"datasetVersion":"2026-08-28T11:17:15.048Z"}