{"record":{"id":"ddc004d0baf2c188","repo":"nathanmarz/storm","slug":"failed-to-get-metadata-for-a-transaction","errorCode":null,"errorMessage":"Failed to get metadata for a transaction","messagePattern":"Failed to get metadata for a transaction","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"storm-core/src/jvm/backtype/storm/transactional/TransactionalSpoutCoordinator.java","lineNumber":160,"sourceCode":"            _collector.emit(TRANSACTION_COMMIT_STREAM_ID, new Values(maybeCommit.attempt), maybeCommit.attempt);\n        }\n        \n        try {\n            if(_activeTx.size() < _maxTransactionActive) {\n                BigInteger curr = _currTransaction;\n                for(int i=0; i<_maxTransactionActive; i++) {\n                    if((_coordinatorState.hasCache(curr) || _coordinator.isReady())\n                            && !_activeTx.containsKey(curr)) {\n                        TransactionAttempt attempt = new TransactionAttempt(curr, _rand.nextLong());\n                        Object state = _coordinatorState.getState(curr, _initializer);\n                        _activeTx.put(curr, new TransactionStatus(attempt));\n                        _collector.emit(TRANSACTION_BATCH_STREAM_ID, new Values(attempt, state, previousTransactionId(_currTransaction)), attempt);\n                    }\n                    curr = nextTransactionId(curr);\n                }\n            }     \n        } catch(FailedException e) {\n            LOG.warn(\"Failed to get metadata for a transaction\", e);\n        }\n    }\n\n    @Override\n    public Map<String, Object> getComponentConfiguration() {\n        Config ret = new Config();\n        ret.setMaxTaskParallelism(1);\n        return ret;\n    }\n    \n    private static enum AttemptStatus {\n        PROCESSING,\n        PROCESSED,\n        COMMITTING\n    }\n    \n    private static class TransactionStatus {\n        TransactionAttempt attempt;","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/nathanmarz/storm/blob/cdb116e942666973bc4eaa0df098d5bab82739e7/storm-core/src/jvm/backtype/storm/transactional/TransactionalSpoutCoordinator.java#L142-L178","documentation":"TransactionalSpoutCoordinator.sync() catches FailedException thrown while obtaining transaction metadata from the coordinator and logs 'Failed to get metadata for a transaction'. Unlike the batch executor, it only logs — no tuple is failed — so sync aborts for this pass and the coordinator simply retries on the next nextTuple/ack/fail driven sync. A persistent underlying failure means no new transaction batches are emitted.","triggerScenarios":"The coordinator's getMetadata/initializeTransaction path throws FailedException (e.g. metadata store read failure) while sync() is advancing _currTransaction and emitting TRANSACTION_BATCH_STREAM tuples; called from nextTuple, ack, or fail.","commonSituations":"Zookeeper or external metadata store unreachable; coordinator code throwing FailedException for transient backend errors; network partition between the coordinator task and its metadata backend; corrupted coordinator state causing repeated metadata fetch failures.","solutions":["Inspect the logged FailedException stack trace to find the metadata source that failed (usually ZooKeeper or a custom store).","Restore/verify the metadata backend (ZK quorum health, connectivity, permissions).","Add defensive retry/backoff in getMetadata for transient backend errors so sync can recover on its own.","Check the coordinator's state in ZooKeeper for corruption; reset transaction state if safe.","If failures persist, restart the coordinator task / topology after fixing the backend."],"exampleFix":"// before\npublic Map<String, Object> getMetadata(TransactionAttempt tx) {\n    return zkClient.readData(path); // transient ZK error -> FailedException each sync\n}\n// after\npublic Map<String, Object> getMetadata(TransactionAttempt tx) {\n    for (int i = 0; i < 3; i++) {\n        try { return zkClient.readData(path); }\n        catch (Exception e) { sleep(backoff(i)); }\n    }\n    throw new FailedException(\"metadata unavailable after retries\");\n}","handlingStrategy":"retry","validationCode":"// before relying on coordinator metadata, verify the backend is reachable\nif (!zkClientExists(coordinatorStatePath)) { throw new IllegalStateException(\"coordinator metadata store unavailable: \" + coordinatorStatePath); }","typeGuard":null,"tryCatchPattern":"public Map<String,Object> getMetadata(TransactionAttempt tx) {\n    try {\n        return readMetadata(tx);\n    } catch (TransientBackendException e) {\n        throw new FailedException(e); // sync() logs and retries on next nextTuple/ack/fail\n    }\n}","preventionTips":["Ensure the metadata store (usually ZooKeeper) is healthy and sized for the cluster.","Implement bounded retry/backoff inside coordinator getMetadata.","Alert on repeated 'Failed to get metadata' warnings — they mean no new batches are emitted.","Keep coordinator transaction state clean; verify after hard crashes or manual ZK edits."],"tags":["storm","transactional-spout","metadata","coordinator","zookeeper"],"backgroundTag":"network-request-failed","analyzedSha":"cdb116e942666973bc4eaa0df098d5bab82739e7","analyzedAt":"2026-09-12T14:30:00.714Z","contentChangedAt":"2026-09-12T14:30:00.714Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}