{"record":{"id":"8a4d4f33624a803e","repo":"juicedata/juicefs","slug":"failed-to-clean-up-sid-d-8a4d4f","errorCode":null,"errorMessage":"failed to clean up sid %d","messagePattern":"failed to clean up sid (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/meta/sql.go","lineNumber":3192,"sourceCode":"\tvar sus []sustained\n\terr = m.simpleTxn(Background(), func(ses *xorm.Session) error {\n\t\tsus = nil\n\t\treturn ses.Find(&sus, &sustained{Sid: sid})\n\t})\n\tif err != nil {\n\t\tlogger.Warnf(\"Scan sustained with sid %d: %s\", sid, err)\n\t\tfail = true\n\t} else {\n\t\tfor _, su := range sus {\n\t\t\tif err = m.doDeleteSustainedInode(sid, su.Inode); err != nil {\n\t\t\t\tlogger.Warnf(\"Delete sustained inode %d of sid %d: %s\", su.Inode, sid, err)\n\t\t\t\tfail = true\n\t\t\t}\n\t\t}\n\t}\n\n\tif fail {\n\t\treturn fmt.Errorf(\"failed to clean up sid %d\", sid)\n\t} else {\n\t\treturn m.txn(func(s *xorm.Session) error {\n\t\t\tvar deleted bool\n\t\t\tif n, err := s.Delete(&session2{Sid: sid}); err != nil {\n\t\t\t\treturn err\n\t\t\t} else if n == 1 {\n\t\t\t\tdeleted = true\n\t\t\t}\n\t\t\tok, err := s.IsTableExist(&session{})\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tif ok {\n\t\t\t\tn, err := s.Delete(&session{Sid: sid})\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\tdeleted = deleted || n == 1","sourceCodeStart":3174,"sourceCodeEnd":3210,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/meta/sql.go#L3174-L3210","documentation":"Returned by dbMeta.doCleanStaleSession when any step of releasing a dead session's resources failed: deleting its flock/plock rows, scanning its sustained inodes, or deleting sustained inodes. When `fail` is set, the session row itself is deliberately NOT deleted so cleanup can be retried; only when everything succeeds is the session record removed.","triggerScenarios":"Stale-session cleanup (heartbeat expiry, `juicefs gc`) on a dead sid while the database is degraded: connection failures deleting flock/plock, read errors scanning sustained, or doDeleteSustainedInode failing (e.g. chunk deletion errors, txn conflicts) for any of the session's open inodes.","commonSituations":"Metadata DB flapping during a mass client crash (network cut kills many clients at once, cleanup storms the DB); blob store (object storage) errors while deleting chunks of sustained inodes; lock-wait timeouts on busy tables during cleanup.","solutions":["Check the preceding Warnf log lines ('Delete flock/plock with sid', 'Scan sustained with sid', 'Delete sustained inode ... of sid') to find which step failed and fix its root cause.","Verify metadata DB and object storage health; cleanup is retried on the next scan once healthy.","If a specific sustained inode keeps failing, run `juicefs fsck` on that inode and resolve data/metadata inconsistency.","Reduce cleanup pressure (stagger client restarts) to avoid lock timeouts during mass cleanup."],"exampleFix":"// before: cleanup logs 'Delete sustained inode 100 of sid 42: ...' then \"failed to clean up sid 42\"; session row kept\n// after: fix object storage access, let the next cleanup pass succeed\njuicefs gc sqlite3://test.db   # re-run after restoring DB/object-store health; session2 row is then deleted","handlingStrategy":"retry","validationCode":"// preflight: metadata DB and object storage reachable before triggering cleanup\nif err := db.Ping(); err != nil { return err }\nif err := objStore.Head(ctx, \"__probe\"); err != nil { return err }","typeGuard":null,"tryCatchPattern":"if err := cleanupStaleSessions(); err != nil {\n    if strings.Contains(err.Error(), \"failed to clean up sid\") {\n        logger.Warnf(\"cleanup deferred, will retry next scan: %v\", err)\n        time.Sleep(time.Minute)\n        return cleanupStaleSessions() // session row was kept, retry is safe\n    }\n    return err\n}","preventionTips":["Ensure object storage credentials/health before running `juicefs gc` or letting cleanup run.","Stagger mass client shutdowns to avoid cleanup storms against the DB.","Investigate per-inode Warnf logs when the same sid repeatedly fails cleanup."],"tags":["session","cleanup","database","metadata"],"backgroundTag":"database-write-failed","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}