{"record":{"id":"2d2bc6eeb03c6d71","repo":"crowdsecurity/crowdsec","slug":"exceeded-d-busy-retries","errorCode":null,"errorMessage":"exceeded %d busy retries","messagePattern":"exceeded (.+?) busy retries","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/database/alerts.go","lineNumber":582,"sourceCode":"\nfunc retryOnBusy(fn func() error) error {\n\tfor retry := range maxLockRetries {\n\t\terr := fn()\n\t\tif err == nil {\n\t\t\treturn nil\n\t\t}\n\n\t\tif IsSqliteBusyError(err) {\n\t\t\tlog.Warningf(\"while updating decisions, sqlite3.ErrBusy: %s, retry %d of %d\", err, retry, maxLockRetries)\n\t\t\ttime.Sleep(1 * time.Second)\n\n\t\t\tcontinue\n\t\t}\n\n\t\treturn err\n\t}\n\n\treturn fmt.Errorf(\"exceeded %d busy retries\", maxLockRetries)\n}\n\nfunc (c *Client) saveAlerts(ctx context.Context, client *ent.Client, batch []alertCreatePlan) ([]string, error) {\n\tif len(batch) == 0 {\n\t\tlog.Warningf(\"no alerts to create, discarded?\")\n\t\treturn nil, nil\n\t}\n\n\t// extract builders in the same order\n\tbuilders := make([]*ent.AlertCreate, len(batch))\n\tfor i := range batch {\n\t\tif batch[i].builder == nil {\n\t\t\treturn nil, fmt.Errorf(\"nil alert builder at index %d\", i)\n\t\t}\n\n\t\tbuilders[i] = batch[i].builder\n\t}\n","sourceCodeStart":564,"sourceCodeEnd":600,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/database/alerts.go#L564-L600","documentation":"retryOnBusy retried the wrapped database operation maxLockRetries times because SQLite kept returning SQLITE_BUSY (database is locked), and every retry still hit the busy condition. This indicates sustained lock contention on the local SQLite database — another process (cscli, a second crowdsec, LAPI) holds a write lock longer than the retry window (1s between attempts).","triggerScenarios":"Concurrent writes to the same SQLite DB while alerts/decisions are being saved: running cscli commands while crowdsec is writing, multiple crowdsec instances sharing one DB, long transactions from other clients, or a slow/hung writer holding the lock.","commonSituations":"Running `cscli decisions list/add/delete` at the same time the alert flush is saving a large batch; two crowdsec processes pointed at the same data dir by mistake; backups or other tools holding a read lock on crowdsec.db; disk I/O stall making transactions slow.","solutions":["Ensure only one crowdsec process uses the database (check for duplicate agents/LAPI sharing one data dir)","Avoid running heavy cscli writes concurrently with alert ingestion; retry the failed operation once contention subsides","Find the lock holder: `fuser`/`lsof` on the sqlite db file, or check for stuck cscli/backup jobs","If contention is chronic, switch the database backend to Postgres/MySQL, which handles concurrency far better than SQLite","Check I/O health (disk saturation, NFS mounts) — slow storage makes SQLite transactions hold locks longer"],"exampleFix":"// before (diagnosis)\ncrowdsec -c /etc/crowdsec/local/... -- db_config.sqlite\n// after (mitigation: dedicated connection tuning)\n# in crowdsec config: use postgres for concurrent writers\ndb_config:\n  type: postgresql\n  host: localhost\n  ...","handlingStrategy":"retry","validationCode":"// caller-side: check DB writability and single-instance before bulk writes\nif err := db.PingContext(ctx); err != nil { return err }","typeGuard":null,"tryCatchPattern":"if err := retryOnBusy(op); err != nil {\n    if strings.Contains(err.Error(), \"exceeded\") {\n        // all busy retries exhausted: back off longer or move to a client-server DB\n    }\n    return err\n}","preventionTips":["Never run two crowdsec instances against one SQLite file","Serialize cscli write commands with ingestion (or accept occasional retries)","Consider postgresql/mysql for multi-writer deployments","Keep the DB on fast local storage; avoid NFS"],"tags":["sqlite","database","lock-contention","concurrency"],"backgroundTag":"database-locked","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}