jaegertracing/jaeger · error

failed to acquire resource lock due to cassandra error: %w

Error message

failed to acquire resource lock due to cassandra error: %w

What it means

Lock.Acquire performs a lightweight-transaction INSERT (IF NOT EXISTS) into the leases table to claim a distributed lock. This error wraps any Cassandra driver failure during that LWT — it means the CAS never reached a decision, not that the lock is merely held. The wrapped gocql error carries the actual cause.

Source

Thrown at internal/storage/distributedlock/cassandra/lock.go:49

// NewLock creates a new instance of a distributed locking mechanism based off Cassandra.
func NewLock(session cassandra.Session, tenantID string) *Lock {
	return &Lock{
		session:  session,
		tenantID: tenantID,
	}
}

// Acquire acquires a lease around a given resource. NB. Cassandra only allows ttl of seconds granularity
func (l *Lock) Acquire(resource string, ttl time.Duration) (bool, error) {
	if ttl == 0 {
		ttl = defaultTTL
	}
	ttlSec := int(ttl.Seconds())
	var name, owner string
	applied, err := l.session.Query(cqlInsertLock, resource, l.tenantID, ttlSec).ScanCAS(&name, &owner)
	if err != nil {
		return false, fmt.Errorf("failed to acquire resource lock due to cassandra error: %w", err)
	}
	if applied {
		// The lock was successfully created
		return true, nil
	}
	if owner == l.tenantID {
		// This host already owns the lock, extend the lease
		if err = l.extendLease(resource, ttl); err != nil {
			return false, fmt.Errorf("failed to extend lease on resource lock: %w", err)
		}
		return true, nil
	}
	return false, nil
}

// Forfeit forfeits an existing lease around a given resource.
func (l *Lock) Forfeit(resource string) (bool, error) {
	var name, owner string

View on GitHub (pinned to 806f444784)

Solutions

1. Inspect the wrapped driver error (errors.Unwrap / %v) to see the exact Cassandra failure.
2. Ensure the Jaeger Cassandra schema (leases table) is initialized (run the schema creation job).
3. Check cluster connectivity: nodetool status, contact points, ports, auth credentials.
4. If LWT timeouts occur, check SERIAL consistency settings and network latency between client and cluster.
5. Retry Acquire with backoff — LWT timeouts are often transient.

Example fix

// before
acquired, err := lock.Acquire("index-cleaner", time.Minute)
if err != nil {
    return err
}
// after
acquired, err := lock.Acquire("index-cleaner", time.Minute)
if err != nil {
    return retry.WithBackoff(ctx, func() error {
        var rerr error
        acquired, rerr = lock.Acquire("index-cleaner", time.Minute)
        return rerr
    }, 3)
}

// verify schema exists before acquiring
iter := session.Query("SELECT name FROM system_schema.tables WHERE keyspace_name = ? AND table_name = 'leases'", keyspace).Iter()
if iter.NumRows() == 0 {
    return errors.New("leases table missing; run schema init")
}
iter.Close()

var acquired bool
err := retry.WithBackoff(ctx, 3, func() error {
    var rerr error
    acquired, rerr = lock.Acquire(resource, ttl)
    return rerr
})
if err != nil {
    return fmt.Errorf("cannot acquire lock %s: %w", resource, err)
}

When it happens

Trigger: Calling Acquire(resource, ttl) when the Cassandra session cannot execute the cqlInsertLock ScanCAS: cluster unreachable, keyspace/leases table missing, LWT timeout (NoHostAvailable, timeout with SERIAL consistency), or invalid TTL values rejected by Cassandra.

Common situations: leases table not created (schema init skipped); Cassandra down or flapping network during schema-migration jobs that use this lock; LWT timeouts under multi-DC latency with SERIAL consistency; misconfigured contact points.

Related errors

• failed to extend lease on resource lock: %w
• failed to forfeit resource lock due to cassandra error: %w
• invalid version
• failed to Exec query '%s': %w
• failed to forfeit resource lock: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/5da199f4e0ed2403. Report an issue: GitHub.