{"record":{"id":"0dc1188787da06d5","repo":"FuelLabs/fuel-core","slug":"block-production-timed-out","errorCode":null,"errorMessage":"Block production timed out","messagePattern":"Block production timed out","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/services/consensus_module/poa/src/service.rs","lineNumber":307,"sourceCode":"    C: GetTime,\n    RS: WaitForReadySignal,\n    RP: BlockReconciliationReadPort,\n{\n    // Request the block producer to make a new block, and return it when ready\n    async fn signal_produce_block(\n        &self,\n        height: BlockHeight,\n        block_time: Tai64,\n        source: TransactionsSource,\n        deadline: Instant,\n    ) -> anyhow::Result<UncommittedExecutionResult<Changes>> {\n        let future = self\n            .block_producer\n            .produce_and_execute_block(height, block_time, source, deadline);\n\n        let result = tokio::time::timeout(self.production_timeout, future)\n            .await\n            .map_err(|_| anyhow::anyhow!(\"Block production timed out\"))??;\n\n        // In the case if the block production finished before the deadline\n        // we need to wait until the deadline is reached to guarantee\n        // the correct interval between blocks\n        sleep_until(deadline).await;\n\n        Ok(result)\n    }\n\n    pub(crate) async fn produce_next_block(\n        &mut self,\n        deadline: Instant,\n    ) -> anyhow::Result<()> {\n        self.produce_block(\n            self.next_height(),\n            self.next_time(RequestType::Trigger)?,\n            TransactionsSource::TxPool,\n            deadline,","sourceCodeStart":289,"sourceCodeEnd":325,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/services/consensus_module/poa/src/service.rs#L289-L325","documentation":"Thrown by the PoA main task in signal_produce_block: the BlockProducer::produce_and_execute_block future is wrapped in tokio::time::timeout(self.production_timeout, ...) (Config.production_timeout, default 20s). If selecting transactions from the pool plus executing the whole block does not finish within that budget, the future is dropped and this anyhow error aborts the production attempt. The error propagates out of produce_block/produce_next_block and the block for that height is simply not produced.","triggerScenarios":"Trigger-driven production (Trigger::Interval/Instant/Open firing produce_next_block) or manual production where produce_and_execute_block exceeds production_timeout. Concretely: a stuffed txpool with a high block_gas_limit, slow VM execution or storage I/O, DA compression enabled adding latency, or a config where production_timeout was lowered (e.g. to 500ms) while blocks legitimately take seconds.","commonSituations":"Dev/test chains with huge block_gas_limit and a flooded mempool; CI runners or low-spec VMs with slow disks; misconfigured production_timeout copied from another node's config; fuel-core version upgrades that made execution slower (new tx types, more consensus rules).","solutions":["Increase Config.production_timeout (crates/services/consensus_module/poa/src/config.rs:14, default Duration::from_secs(20)) to comfortably exceed worst-case execution time for your block_gas_limit.","Lower block_gas_limit (and block transaction count) so worst-case execution fits inside the timeout.","Profile the producer/executor: check disk latency, executor metrics, and whether skipped-transaction processing dominates; fix the underlying slowness rather than only raising the timeout.","Drain or cap the transaction pool backlog so each block's execution set is bounded.","Treat a single timeout as transient: the next trigger tick retries production at the same height."],"exampleFix":"// before (PoA Config)\nlet config = Config {\n    production_timeout: Duration::from_millis(500),\n    ..Default::default()\n};\n\n// after\nlet config = Config {\n    production_timeout: Duration::from_secs(20),\n    ..Default::default()\n};","handlingStrategy":"retry","validationCode":"// Before starting the service, sanity-check that the timeout can cover worst-case execution\n// for your block gas limit (rough heuristic: measure an empty block first).\nfn validate_production_timeout(config: &fuel_core_poa::Config, worst_case_execution: Duration) -> anyhow::Result<()> {\n    anyhow::ensure!(\n        config.production_timeout > worst_case_execution,\n        \"production_timeout ({:?}) must exceed worst-case execution ({:?})\",\n        config.production_timeout, worst_case_execution\n    );\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"// Timeout is transient (dropped future, no partial commit): match the message, back off, and let the next trigger retry.\nmatch produce_result {\n    Ok(()) => {}\n    Err(err) if err.to_string().contains(\"Block production timed out\") => {\n        tracing::warn!(\"block production timed out; retrying on next trigger\");\n        // do not restart the node; await the next trigger tick / re-issue manual production\n    }\n    Err(err) => return Err(err),\n}","preventionTips":["Size production_timeout against block_gas_limit; bigger blocks need a bigger budget.","Expose metrics for produce_and_execute_block duration and alert when it approaches production_timeout.","Keep the transaction pool backlog bounded so per-block execution time is predictable.","Never lower production_timeout on producing nodes without load-testing first."],"tags":["consensus","poa","timeout","block-production","performance","rust"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}