{"record":{"id":"5da199f4e0ed2403","repo":"jaegertracing/jaeger","slug":"failed-to-acquire-resource-lock-due-to-cassandra-e","errorCode":null,"errorMessage":"failed to acquire resource lock due to cassandra error: %w","messagePattern":"failed to acquire resource lock due to cassandra error: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/distributedlock/cassandra/lock.go","lineNumber":49,"sourceCode":"\n// NewLock creates a new instance of a distributed locking mechanism based off Cassandra.\nfunc NewLock(session cassandra.Session, tenantID string) *Lock {\n\treturn &Lock{\n\t\tsession:  session,\n\t\ttenantID: tenantID,\n\t}\n}\n\n// Acquire acquires a lease around a given resource. NB. Cassandra only allows ttl of seconds granularity\nfunc (l *Lock) Acquire(resource string, ttl time.Duration) (bool, error) {\n\tif ttl == 0 {\n\t\tttl = defaultTTL\n\t}\n\tttlSec := int(ttl.Seconds())\n\tvar name, owner string\n\tapplied, err := l.session.Query(cqlInsertLock, resource, l.tenantID, ttlSec).ScanCAS(&name, &owner)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to acquire resource lock due to cassandra error: %w\", err)\n\t}\n\tif applied {\n\t\t// The lock was successfully created\n\t\treturn true, nil\n\t}\n\tif owner == l.tenantID {\n\t\t// This host already owns the lock, extend the lease\n\t\tif err = l.extendLease(resource, ttl); err != nil {\n\t\t\treturn false, fmt.Errorf(\"failed to extend lease on resource lock: %w\", err)\n\t\t}\n\t\treturn true, nil\n\t}\n\treturn false, nil\n}\n\n// Forfeit forfeits an existing lease around a given resource.\nfunc (l *Lock) Forfeit(resource string) (bool, error) {\n\tvar name, owner string","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/distributedlock/cassandra/lock.go#L31-L67","documentation":"Lock.Acquire performs a lightweight-transaction INSERT (IF NOT EXISTS) into the leases table to claim a distributed lock. This error wraps any Cassandra driver failure during that LWT — it means the CAS never reached a decision, not that the lock is merely held. The wrapped gocql error carries the actual cause.","triggerScenarios":"Calling Acquire(resource, ttl) when the Cassandra session cannot execute the cqlInsertLock ScanCAS: cluster unreachable, keyspace/leases table missing, LWT timeout (NoHostAvailable, timeout with SERIAL consistency), or invalid TTL values rejected by Cassandra.","commonSituations":"leases table not created (schema init skipped); Cassandra down or flapping network during schema-migration jobs that use this lock; LWT timeouts under multi-DC latency with SERIAL consistency; misconfigured contact points.","solutions":["Inspect the wrapped driver error (errors.Unwrap / %v) to see the exact Cassandra failure.","Ensure the Jaeger Cassandra schema (leases table) is initialized (run the schema creation job).","Check cluster connectivity: nodetool status, contact points, ports, auth credentials.","If LWT timeouts occur, check SERIAL consistency settings and network latency between client and cluster.","Retry Acquire with backoff — LWT timeouts are often transient."],"exampleFix":"// before\nacquired, err := lock.Acquire(\"index-cleaner\", time.Minute)\nif err != nil {\n    return err\n}\n// after\nacquired, err := lock.Acquire(\"index-cleaner\", time.Minute)\nif err != nil {\n    return retry.WithBackoff(ctx, func() error {\n        var rerr error\n        acquired, rerr = lock.Acquire(\"index-cleaner\", time.Minute)\n        return rerr\n    }, 3)\n}","handlingStrategy":"retry","validationCode":"// verify schema exists before acquiring\niter := session.Query(\"SELECT name FROM system_schema.tables WHERE keyspace_name = ? AND table_name = 'leases'\", keyspace).Iter()\nif iter.NumRows() == 0 {\n    return errors.New(\"leases table missing; run schema init\")\n}\niter.Close()","typeGuard":null,"tryCatchPattern":"var acquired bool\nerr := retry.WithBackoff(ctx, 3, func() error {\n    var rerr error\n    acquired, rerr = lock.Acquire(resource, ttl)\n    return rerr\n})\nif err != nil {\n    return fmt.Errorf(\"cannot acquire lock %s: %w\", resource, err)\n}","preventionTips":["Initialize the Jaeger Cassandra schema (leases table) in every environment.","Give each host a unique tenantID.","Retry LWT timeouts with backoff; they are frequently transient.","Monitor Cassandra SERIAL-consistency latency in multi-DC setups."],"tags":["cassandra","distributed-lock","lwt","storage"],"backgroundTag":"distributed-lock-acquire-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}