{"record":{"id":"93a24663682187e8","repo":"apache/iceberg","slug":"interrupted-finding-locks-to-unlock","errorCode":null,"errorMessage":"Interrupted finding locks to unlock {}.{}","messagePattern":"Interrupted finding locks to unlock (.+?)\\.(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java","lineNumber":423,"sourceCode":"        id = lockId.get();\n      }\n\n      doUnlock(id);\n    } catch (InterruptedException ie) {\n      if (id != null) {\n        // Interrupted unlock. We try to unlock one more time if we have a lockId\n        try {\n          Thread.interrupted(); // Clear the interrupt status flag for now, so we can retry unlock\n          LOG.warn(\"Interrupted unlock we try one more time {}.{}\", databaseName, tableName, ie);\n          doUnlock(id);\n        } catch (Exception e) {\n          LOG.warn(\"Failed to unlock even on 2nd attempt {}.{}\", databaseName, tableName, e);\n        } finally {\n          Thread.currentThread().interrupt(); // Set back the interrupt status\n        }\n      } else {\n        Thread.currentThread().interrupt(); // Set back the interrupt status\n        LOG.warn(\"Interrupted finding locks to unlock {}.{}\", databaseName, tableName, ie);\n      }\n    } catch (Exception e) {\n      LOG.warn(\"Failed to unlock {}.{}\", databaseName, tableName, e);\n    }\n  }\n\n  private void doUnlock(long lockId) throws TException, InterruptedException {\n    metaClients.run(\n        client -> {\n          client.unlock(lockId);\n          return null;\n        });\n  }\n\n  private void acquireJvmLock() {\n    if (jvmLock != null) {\n      throw new IllegalStateException(\n          String.format(\"Cannot call acquireLock twice for %s\", fullName));","sourceCodeStart":405,"sourceCodeEnd":441,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java#L405-L441","documentation":"MetastoreLock.unlock() interrupts the thread waiting on Hive Metastore lock cleanup when the RPC to locate held locks (get_locks) is interrupted. Since the unlock is failing during shutdown/rollback, the library logs a warning, restores the interrupt status, and gives up; the Hive locks may be left orphaned until the metastore's lock timeout reaps them. This is a best-effort path, not a thrown exception.","triggerScenarios":"Thread waiting inside unlock()'s Thrift client call (get_locks via IMetaStoreClient) is interrupted — typically Task.cancel(), query cancellation, or JVM shutdown interrupting the Iceberg commit/abort path that acquired a Hive lock on databaseName.tableName.","commonSituations":"Spark/Flink/Trino query cancellation while a Hive-lock-based Iceberg commit is aborting; executor shutdown during long commits; kill -SIGINT of a job mid-commit.","solutions":["Re-check Hive metastore for orphaned locks (SHOW LOCKS / metastore lock tables) and clear them or let heartbeats/timeouts expire them.","Avoid interrupting threads during commit; cancel at a point between operations, or use a lock manager (e.g. DynamoDbLockManager) that cleans up reliably.","Ensure the metastore client call timeout is shorter than the interrupt source's patience so cleanup completes before cancellation.","Increase hive.txn.timeout / lock reaper settings so orphaned locks do not block subsequent commits."],"exampleFix":"// before\nfuture.cancel(true); // interrupts commit thread mid-unlock, orphans locks\n// after\nfuture.cancel(false); // or await commit completion before cancelling","handlingStrategy":"retry","validationCode":"// before committing, check for pre-existing locks\nList<ShowLocksResponseElement> locks = client.showLocks(new ShowLocksRequest(db, table)).getLocks();\nif (locks.stream().anyMatch(l -> l.getState().equals(\"ACQUIRED\"))) {\n  throw new IllegalStateException(\"Hive lock already held on \" + db + \".\" + table);\n}","typeGuard":null,"tryCatchPattern":"try {\n  commitWithLock();\n} catch (InterruptedException ie) {\n  Thread.currentThread().interrupt();\n  // schedule lock cleanup in a separate, non-interrupted thread\n  cleanupExecutor.submit(() -> unlockQuietly(db, table));\n}","preventionTips":["Don't interrupt threads that are mid-commit; prefer cooperative cancellation checkpoints.","Use a lock manager (DynamoDbLockManager) that supports reliable unlock on cancellation.","Tune hive lock timeout so orphaned locks expire quickly.","Monitor SHOW LOCKS for accumulating ACQUIRED locks on Iceberg tables."],"tags":["hive","locking","interruption","metastore"],"backgroundTag":"request-timeout","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}