{"record":{"id":"aa9c5dfe74426bad","repo":"RocketChat/Rocket.Chat","slug":"failed-to-get-apps-status-from-instance-instance","errorCode":null,"errorMessage":"Failed to get apps status from instance ${instanceId}","messagePattern":"Failed to get apps status from instance (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"apps/meteor/ee/server/local-services/instance/service.ts","lineNumber":239,"sourceCode":"\t\t\tthrow new AppsEngineNoNodesFoundError();\n\t\t}\n\n\t\tconst control: Promise<void>[] = [];\n\t\tconst statusByApp: AppStatusReport = {};\n\n\t\tinstances.forEach((instance) => {\n\t\t\tconst { id: instanceId } = instance;\n\n\t\t\tcontrol.push(\n\t\t\t\t(async () => {\n\t\t\t\t\tconst appsStatus = await this.broker.call<Awaited<ReturnType<(typeof Apps)['getAppsStatusLocal']>>, null>(\n\t\t\t\t\t\t'matrix.getAppsStatus',\n\t\t\t\t\t\tnull,\n\t\t\t\t\t\t{ nodeID: instanceId },\n\t\t\t\t\t);\n\n\t\t\t\t\tif (!appsStatus) {\n\t\t\t\t\t\tthrow new Error(`Failed to get apps status from instance ${instanceId}`);\n\t\t\t\t\t}\n\n\t\t\t\t\tappsStatus.forEach(({ status, appId }) => {\n\t\t\t\t\t\tif (!statusByApp[appId]) {\n\t\t\t\t\t\t\tstatusByApp[appId] = [];\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\tstatusByApp[appId].push({ instanceId, isLocal: instance.local, status });\n\t\t\t\t\t});\n\t\t\t\t})(),\n\t\t\t);\n\t\t});\n\n\t\tawait Promise.all(control);\n\n\t\treturn statusByApp;\n\t}\n}","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/RocketChat/Rocket.Chat/blob/b2c16d5842cbe6b69b59bdf6fc5e5f1afcd1f0b0/apps/meteor/ee/server/local-services/instance/service.ts#L221-L257","documentation":"The Enterprise matrix local service aggregates per-app status across instances by fanning out broker.call('matrix.getAppsStatus', null, { nodeID: instanceId }) to each node. If a target instance responds with a falsy payload (undefined/null/empty), the per-instance task throws `Failed to get apps status from instance ${instanceId}`, failing the whole status collection. A falsy result means that node had no apps status to report - typically its Apps engine is not running or not ready.","triggerScenarios":"Invoking the aggregated apps-status operation in a multi-instance (matrix) deployment while one instance is mid-startup, restarting, running a version without the Apps service, or otherwise answering empty from getAppsStatus. A single bad instance aborts the batch because all per-instance promises are collected and awaited together.","commonSituations":"Rolling restarts where a node has not finished initializing its Apps engine; version skew between nodes in the matrix; a node that just rejoined after a disconnect; instances with zero apps returning undefined instead of an empty array.","solutions":["Inspect the named instanceId: check its logs and broker connectivity, confirm its Apps service finished startup, and restart the node if stuck.","Ensure all instances run compatible versions of the matrix/apps services before aggregating status.","Retry the status operation once the instance is ready; if a node is permanently empty, remove or repair it in the matrix."],"exampleFix":"// before\nconst status = await appsStatusService.getAppsStatus(); // one unready instance -> whole call throws\n\n// after - retry briefly to ride out instance startup\nlet status;\nfor (let attempt = 0; attempt < 3; attempt++) {\n\ttry {\n\t\tstatus = await appsStatusService.getAppsStatus();\n\t\tbreak;\n\t} catch (e: any) {\n\t\tif (!/Failed to get apps status from instance/.test(e?.message) || attempt === 2) throw e;\n\t\tawait sleep(1000 * (attempt + 1)); // instance may still be starting\n\t}\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"let lastErr;\nfor (let attempt = 0; attempt < 3; attempt++) {\n\ttry {\n\t\treturn await getAppsStatusAggregated();\n\t} catch (e: any) {\n\t\tif (!/Failed to get apps status from instance/.test(e?.message)) throw e;\n\t\tlastErr = e;\n\t\tawait sleep(1000 * (attempt + 1)); // instance may still be starting/rejoining\n\t}\n}\nthrow lastErr; // after retries, report the failing instanceId from the message","preventionTips":["Gate instances on Apps-service readiness before they join the matrix.","Keep all nodes on compatible versions during rolling deploys.","Log the instanceId from the message to pinpoint which node returned an empty status."],"tags":["apps-engine","multi-instance","distributed-systems","enterprise"],"backgroundTag":"service-unavailable","analyzedSha":"b2c16d5842cbe6b69b59bdf6fc5e5f1afcd1f0b0","analyzedAt":"2026-08-18T15:26:39.429Z","contentChangedAt":"2026-08-18T15:26:39.429Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}