jaegertracing/jaeger · error

failed to extend lease on resource lock: %w

Error message

failed to extend lease on resource lock: %w

What it means

Lock.Acquire wraps errors from extendLease when the host already owns the lock and attempts to renew its TTL via a conditional UPDATE (IF owner = ?). It can wrap the Cassandra driver error or errLockOwnership ('this host does not own the resource lock') when the CAS condition stopped holding between the insert-check and the update.

Source

Thrown at internal/storage/distributedlock/cassandra/lock.go:58

// Acquire acquires a lease around a given resource. NB. Cassandra only allows ttl of seconds granularity
func (l *Lock) Acquire(resource string, ttl time.Duration) (bool, error) {
	if ttl == 0 {
		ttl = defaultTTL
	}
	ttlSec := int(ttl.Seconds())
	var name, owner string
	applied, err := l.session.Query(cqlInsertLock, resource, l.tenantID, ttlSec).ScanCAS(&name, &owner)
	if err != nil {
		return false, fmt.Errorf("failed to acquire resource lock due to cassandra error: %w", err)
	}
	if applied {
		// The lock was successfully created
		return true, nil
	}
	if owner == l.tenantID {
		// This host already owns the lock, extend the lease
		if err = l.extendLease(resource, ttl); err != nil {
			return false, fmt.Errorf("failed to extend lease on resource lock: %w", err)
		}
		return true, nil
	}
	return false, nil
}

// Forfeit forfeits an existing lease around a given resource.
func (l *Lock) Forfeit(resource string) (bool, error) {
	var name, owner string
	applied, err := l.session.Query(cqlDeleteLock, resource, l.tenantID).ScanCAS(&name, &owner)
	if err != nil {
		return false, fmt.Errorf("failed to forfeit resource lock due to cassandra error: %w", err)
	}
	if applied {
		// The lock was successfully deleted
		return true, nil
	}
	return false, fmt.Errorf("failed to forfeit resource lock: %w", errLockOwnership)

View on GitHub (pinned to 806f444784)

Solutions

1. If the wrapped error is errLockOwnership, treat the lock as lost: release/reacquire or abort the critical section.
2. Retry Acquire after a short backoff — the lease may have been taken over and can become free again after TTL expiry.
3. Give each host a unique tenantID so ownership checks are meaningful.
4. Increase the lease TTL or renew earlier if renewals race with expiry.
5. Check Cassandra connectivity for transient driver failures in the wrapped error.

Example fix

// before
ok, err := lock.Acquire("index-cleaner", time.Minute)
if err != nil {
    return err // may be lease-loss; treating as fatal blocks forever
}
// after
ok, err := lock.Acquire("index-cleaner", time.Minute)
if err != nil {
    if errors.Is(err, errLockOwnership) {
        log.Warn("lost distributed lock; aborting this round")
        return nil
    }
    return err
}

// renew well within the TTL instead of at expiry
renewEvery := ttl / 2
startRenewalLoop(ctx, lock, resource, ttl, renewEvery)

func isLeaseLost(err error) bool { return errors.Is(err, errLockOwnership) }

ok, err := lock.Acquire(resource, ttl)
if err != nil {
    if errors.Is(err, errLockOwnership) {
        // lease expired and was taken over; abort or reacquire later
        return ErrLockLost
    }
    return fmt.Errorf("transient lock failure: %w", err)
}

When it happens

Trigger: Calling Acquire while this tenant already owns the lock and either (a) the cqlUpdateLock ScanCAS fails at the driver level (connectivity, timeout), or (b) the lease row's owner no longer matches this tenant (lease expired and was re-acquired by someone else), returning errLockOwnership.

Common situations: Long GC pause or network partition causing the lease TTL to lapse before renewal; clock skew shrinking the effective TTL; two instances sharing the same tenantID racing on renewal.

Related errors

• failed to acquire resource lock due to cassandra error: %w
• failed to forfeit resource lock due to cassandra error: %w
• failed to forfeit resource lock: %w
• trace_ttl can either be 0 or greater than or equal to 1 seco
• dependencies_ttl can either be 0 or greater than or equal to

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/e956b6cb2df709b1. Report an issue: GitHub.