nathanmarz/storm · error · RuntimeException

A single worker should have 1 SystemBolt instance.

Error message

A single worker should have 1 SystemBolt instance.

What it means

SystemBolt is a built-in bolt that should exist exactly once per worker; a static flag _prepareWasCalled detects a second prepare() call in distributed mode and throws. This usually means the worker spawned more than one SystemBolt instance — typically because the topology or its config registers the __system bolt multiple times (e.g. duplicate builtin metric registrations) rather than user code calling prepare directly.

Solutions

1. Inspect the topology definition and remove duplicate declarations of the __system bolt (it should appear once per topology).
2. Check Config.TOPOLOGY_BUILTIN_METRICS_BUCKET_SIZE_SECS and metrics registration code for logic that adds __system more than once.
3. If using a wrapper/test harness, ensure it does not reuse the same SystemBolt class across topologies in one worker; in local mode this check is skipped by design.
4. Compare against the stock default topology config; restore the default builtin-metrics registration if it was customized.

Example fix

// before
topologyBuilder.setBolt("__system", new SystemBolt(), 1);
topologyBuilder.setBolt("__system", new SystemBolt(), 1); // duplicate component name/instance
// after
topologyBuilder.setBolt("__system", new SystemBolt(), 1); // only once per topology

// Before submitting, ensure __system is declared at most once in the topology
long sysCount = topology.getBolts().keySet().stream()
    .filter(n -> n.equals("__system")).count();
if (sysCount > 1) throw new IllegalStateException("__system bolt declared " + sysCount + " times");

// Wrap topology submission on a real cluster
try {
    stormSubmitter.submitTopology(name, conf, topology);
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().contains("1 SystemBolt instance")) {
        // topology registers __system more than once; fix topology definition and resubmit
    } else throw e;
}

When it happens

Trigger: In distributed (non-'local') cluster mode, SystemBolt.prepare is invoked a second time within the same worker JVM, i.e. two SystemBolt instances are created for one worker (duplicate __system component registration or a Storm/config issue causing double instantiation).

Common situations: Running on a real cluster (not LocalCluster) after customizing topology metrics consumer/builtin metric settings that reference __system twice; topology definitions built programmatically with duplicated bolt declarations; Storm version upgrades where worker topology-loading logic changed.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Could not instantiate a class listed in config under section
• Non-system tuples should never be sent to __system bolt.
• MeanReducer::reduce called with unsupported input type
• No output fields defined for component:stream
• TopologyContext.registerMetric can only be called from…

AI-assisted analysis of nathanmarz/storm@cdb116e942 (2026-09-12). Data as JSON: /api/errors/bf84fa95584348d8. Report an issue: GitHub.

Appendix: source

Thrown at storm-core/src/jvm/backtype/storm/metric/SystemBolt.java:95

            Long collectionTimeP = _gcBean.getCollectionTime();

            Map ret = null;
            if(_collectionCount!=null && _collectionTime!=null) {
                ret = new HashMap();
                ret.put("count", collectionCountP - _collectionCount);
                ret.put("timeMs", collectionTimeP - _collectionTime);
            }

            _collectionCount = collectionCountP;
            _collectionTime = collectionTimeP;
            return ret;
        }
    }

    @Override
    public void prepare(final Map stormConf, TopologyContext context, OutputCollector collector) {
        if(_prepareWasCalled && !"local".equals(stormConf.get(Config.STORM_CLUSTER_MODE))) {
            throw new RuntimeException("A single worker should have 1 SystemBolt instance.");
        }
        _prepareWasCalled = true;

        int bucketSize = RT.intCast(stormConf.get(Config.TOPOLOGY_BUILTIN_METRICS_BUCKET_SIZE_SECS));

        final RuntimeMXBean jvmRT = ManagementFactory.getRuntimeMXBean();

        context.registerMetric("uptimeSecs", new IMetric() {
            @Override
            public Object getValueAndReset() {
                return jvmRT.getUptime()/1000.0;
            }
        }, bucketSize);

        context.registerMetric("startTimeSecs", new IMetric() {
            @Override
            public Object getValueAndReset() {
                return jvmRT.getStartTime()/1000.0;

View on GitHub (pinned to cdb116e942)