{"record":{"id":"1422ffac1aa2ff73","repo":"abhigyanpatwari/GitNexus","slug":"worker-workerindex-parse-job-idle-timeout-spli","errorCode":null,"errorMessage":"Worker ${workerIndex} parse job idle timeout. Splitting into ${first.items.length}/${second.items.length} item jobs.","messagePattern":"Worker (.+?) parse job idle timeout\\. Splitting into (.+?)/(.+?) item jobs\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"gitnexus/src/core/ingestion/workers/worker-pool.ts","lineNumber":1900,"sourceCode":"            items: firstItems,\n            estimatedBytes: firstItems.reduce((sum, item) => sum + estimateItemBytes(item), 0),\n            attempt: job.attempt,\n            splitDepth: job.splitDepth + 1,\n            chunkHash: job.chunkHash,\n            timeoutMs: nextTimeout,\n            cumulativeTimeoutMs: nextCumulative,\n          };\n          const second: WorkerJob<TInput> = {\n            startIndex: job.startIndex + midpoint,\n            items: secondItems,\n            estimatedBytes: secondItems.reduce((sum, item) => sum + estimateItemBytes(item), 0),\n            attempt: job.attempt,\n            splitDepth: job.splitDepth + 1,\n            chunkHash: job.chunkHash,\n            timeoutMs: nextTimeout,\n            cumulativeTimeoutMs: nextCumulative,\n          };\n          logger.warn(\n            {\n              workerIndex,\n              timeoutSec: job.timeoutMs / 1000,\n              items: job.items.length,\n              estimatedBytes: job.estimatedBytes,\n              lastProgress,\n              firstSplitItems: first.items.length,\n              secondSplitItems: second.items.length,\n              nextTimeoutSec: nextTimeout / 1000,\n            },\n            `Worker ${workerIndex} parse job idle timeout. Splitting into ${first.items.length}/${second.items.length} item jobs.`,\n          );\n          // Preserve intuitive retry order; final result order is still enforced by startIndex sort.\n          jobs.unshift(first, second);\n          return { kind: 'retry' };\n        }\n\n        const nextAttempt = job.attempt + 1;","sourceCodeStart":1882,"sourceCodeEnd":1918,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/0d1aed942f0e8b5d3bac27519fff441aceea722d/gitnexus/src/core/ingestion/workers/worker-pool.ts#L1882-L1918","documentation":"On an idle timeout for a multi-item job, the pool bisects the job at its midpoint into two smaller jobs (inheriting attempt count, bumping splitDepth, each with recomputed byte estimates and the next backoff timeout) and requeues them at the front. The warn records the original timeout/size plus both halves' item counts so operators can watch the bisection converge on the pathological item.","triggerScenarios":"A sub-batch with multiple items going idle-silent past its timeout: the job splits into first/second WorkerJob halves and jobs.unshift(first, second) requeues them; final result order stays correct via the startIndex sort.","commonSituations":"One heavy file inside a mixed sub-batch stalling the whole batch; timeouts calibrated for typical files on machines slower than expected; large values of subBatchMaxBytes packing many big files into one job.","solutions":["Let the split converge — each bisection isolates the slow item; watch the warn's first/secondSplitItems counts narrow toward 1","Reduce GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES (or subBatchSize) so big files land in smaller jobs from the start","Raise GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS if your hardware makes the initial budget unrealistic","If splitting always isolates the same file, exclude or shrink that file — the single-item retry/quarantine path is next for it"],"exampleFix":"# before\nexport GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=   # big files batched together → split warns\n# after\nexport GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=1048576   # 1MB sub-batches, fewer splits","handlingStrategy":"retry","validationCode":"// Prevent avoidable splits by sizing sub-batches to your file mix:\nif (!process.env.GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES) {\n  process.env.GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES = String(1024 * 1024); // 1MB\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat repeated split warns as a sizing signal — convergence wastes wall-clock even when it succeeds","Lower sub-batch byte caps on repos with a few huge files among many small ones","Note the item counts narrowing to 1/1: the isolated item is your next exclusion or timeout candidate"],"tags":["worker-pool","timeout","job-splitting","bisection","parse-jobs"],"backgroundTag":"job-timeout-split","analyzedSha":"0d1aed942f0e8b5d3bac27519fff441aceea722d","analyzedAt":"2026-08-20T23:29:22.980Z","contentChangedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}