apache/iceberg · warning

Failed to unlock even on 2nd attempt

Error message

Failed to unlock even on 2nd attempt {}.{}

What it means

During MetastoreLock.unlock, if the thread was interrupted and an unlock retry (using the known lockId) also fails, the code logs this warning and finally restores the thread's interrupt status. It indicates a lock may have been left behind in HMS after two failed unlock attempts.

Solutions

1. Check HMS for leaked locks (SHOW LOCKS / agentInfo) and manually unlock if present
2. Avoid cancelling tasks during commit; drain executors gracefully
3. Retry the whole operation — commit path will re-acquire locks appropriately
4. Investigate root interrupt source (shutdown hooks, timeouts) if recurring

Example fix

// before
executor.shutdownNow(); // interrupts commit threads mid-unlock
// after
executor.shutdown();
if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {
  executor.shutdownNow(); // interrupt only after graceful drain
}

try { operation.cancel(false); } catch (InterruptedException ie) { Thread.currentThread().interrupt(); /* preserve status */ }

When it happens

Trigger: Thread interrupted while unlocking (executor shutdown, task cancellation) AND the immediate doUnlock retry throws again (HMS unreachable, TException, InterruptedException again).

Common situations: Cancelling Spark/Flink jobs mid-commit; pod termination in Kubernetes; thread pools shutting down with in-flight commits.

Related errors

• Could not find lock with HMSClient
• Failed to create lock
• Failed to unlock .
• Hive lock heartbeat thread not active
• Interrupted during commit

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1d6e1195de8d1ead. Report an issue: GitHub.

Appendix: source

Thrown at hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java:417

          id = lockInfo.lockId;
        } else {
          LOG.warn("Could not find lock with HMSClient {}", HiveVersion.current());
          return;
        }
      } else {
        id = lockId.get();
      }

      doUnlock(id);
    } catch (InterruptedException ie) {
      if (id != null) {
        // Interrupted unlock. We try to unlock one more time if we have a lockId
        try {
          Thread.interrupted(); // Clear the interrupt status flag for now, so we can retry unlock
          LOG.warn("Interrupted unlock we try one more time {}.{}", databaseName, tableName, ie);
          doUnlock(id);
        } catch (Exception e) {
          LOG.warn("Failed to unlock even on 2nd attempt {}.{}", databaseName, tableName, e);
        } finally {
          Thread.currentThread().interrupt(); // Set back the interrupt status
        }
      } else {
        Thread.currentThread().interrupt(); // Set back the interrupt status
        LOG.warn("Interrupted finding locks to unlock {}.{}", databaseName, tableName, ie);
      }
    } catch (Exception e) {
      LOG.warn("Failed to unlock {}.{}", databaseName, tableName, e);
    }
  }

  private void doUnlock(long lockId) throws TException, InterruptedException {
    metaClients.run(
        client -> {
          client.unlock(lockId);
          return null;
        });

View on GitHub (pinned to 86d9c8fc54)