{"record":{"id":"e956b6cb2df709b1","repo":"jaegertracing/jaeger","slug":"failed-to-extend-lease-on-resource-lock-w","errorCode":null,"errorMessage":"failed to extend lease on resource lock: %w","messagePattern":"failed to extend lease on resource lock: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/distributedlock/cassandra/lock.go","lineNumber":58,"sourceCode":"// Acquire acquires a lease around a given resource. NB. Cassandra only allows ttl of seconds granularity\nfunc (l *Lock) Acquire(resource string, ttl time.Duration) (bool, error) {\n\tif ttl == 0 {\n\t\tttl = defaultTTL\n\t}\n\tttlSec := int(ttl.Seconds())\n\tvar name, owner string\n\tapplied, err := l.session.Query(cqlInsertLock, resource, l.tenantID, ttlSec).ScanCAS(&name, &owner)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to acquire resource lock due to cassandra error: %w\", err)\n\t}\n\tif applied {\n\t\t// The lock was successfully created\n\t\treturn true, nil\n\t}\n\tif owner == l.tenantID {\n\t\t// This host already owns the lock, extend the lease\n\t\tif err = l.extendLease(resource, ttl); err != nil {\n\t\t\treturn false, fmt.Errorf(\"failed to extend lease on resource lock: %w\", err)\n\t\t}\n\t\treturn true, nil\n\t}\n\treturn false, nil\n}\n\n// Forfeit forfeits an existing lease around a given resource.\nfunc (l *Lock) Forfeit(resource string) (bool, error) {\n\tvar name, owner string\n\tapplied, err := l.session.Query(cqlDeleteLock, resource, l.tenantID).ScanCAS(&name, &owner)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to forfeit resource lock due to cassandra error: %w\", err)\n\t}\n\tif applied {\n\t\t// The lock was successfully deleted\n\t\treturn true, nil\n\t}\n\treturn false, fmt.Errorf(\"failed to forfeit resource lock: %w\", errLockOwnership)","sourceCodeStart":40,"sourceCodeEnd":76,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/distributedlock/cassandra/lock.go#L40-L76","documentation":"Lock.Acquire wraps errors from extendLease when the host already owns the lock and attempts to renew its TTL via a conditional UPDATE (IF owner = ?). It can wrap the Cassandra driver error or errLockOwnership ('this host does not own the resource lock') when the CAS condition stopped holding between the insert-check and the update.","triggerScenarios":"Calling Acquire while this tenant already owns the lock and either (a) the cqlUpdateLock ScanCAS fails at the driver level (connectivity, timeout), or (b) the lease row's owner no longer matches this tenant (lease expired and was re-acquired by someone else), returning errLockOwnership.","commonSituations":"Long GC pause or network partition causing the lease TTL to lapse before renewal; clock skew shrinking the effective TTL; two instances sharing the same tenantID racing on renewal.","solutions":["If the wrapped error is errLockOwnership, treat the lock as lost: release/reacquire or abort the critical section.","Retry Acquire after a short backoff — the lease may have been taken over and can become free again after TTL expiry.","Give each host a unique tenantID so ownership checks are meaningful.","Increase the lease TTL or renew earlier if renewals race with expiry.","Check Cassandra connectivity for transient driver failures in the wrapped error."],"exampleFix":"// before\nok, err := lock.Acquire(\"index-cleaner\", time.Minute)\nif err != nil {\n    return err // may be lease-loss; treating as fatal blocks forever\n}\n// after\nok, err := lock.Acquire(\"index-cleaner\", time.Minute)\nif err != nil {\n    if errors.Is(err, errLockOwnership) {\n        log.Warn(\"lost distributed lock; aborting this round\")\n        return nil\n    }\n    return err\n}","handlingStrategy":"try-catch","validationCode":"// renew well within the TTL instead of at expiry\nrenewEvery := ttl / 2\nstartRenewalLoop(ctx, lock, resource, ttl, renewEvery)","typeGuard":"func isLeaseLost(err error) bool { return errors.Is(err, errLockOwnership) }","tryCatchPattern":"ok, err := lock.Acquire(resource, ttl)\nif err != nil {\n    if errors.Is(err, errLockOwnership) {\n        // lease expired and was taken over; abort or reacquire later\n        return ErrLockLost\n    }\n    return fmt.Errorf(\"transient lock failure: %w\", err)\n}","preventionTips":["Set TTLs comfortably longer than the critical section; renew at TTL/2 intervals.","Use unique tenantIDs per instance to detect real ownership races.","Avoid long GC pauses inside the locked section.","Handle ErrLockLost as an expected, non-fatal outcome."],"tags":["cassandra","distributed-lock","lease-expiry","lwt"],"backgroundTag":"distributed-lock-lease-lost","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}