{"record":{"id":"aab466e9fd589db1","repo":"SonarSource/sonarqube","slug":"elasticsearch-nodes-have-critically-low-disk-space","errorCode":null,"errorMessage":"Elasticsearch nodes have critically low disk space","messagePattern":"Elasticsearch nodes have critically low disk space","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/sonar-webserver-monitoring/src/main/java/org/sonar/server/monitoring/ElasticSearchMetricTask.java","lineNumber":84,"sourceCode":"  private void updateElasticSearchHealthStatus(@Nullable final NodeStatsResponse nodeStatsResponse) {\n    try {\n      HealthStatus esStatus = esClient.clusterHealthV2(req -> req).status();\n\n      long readOnlyIndicesCount = updateAndGetReadOnlyIndicesCount();\n      final double thresholdPercent = config.getDouble(DISK_SPACE_THRESHOLD_PROPERTY)\n        .orElse(DEFAULT_DISK_SPACE_THRESHOLD_PERCENT);\n      double maxDiskUsagePercent = updateAndGetMaxDiskUsagePercent(nodeStatsResponse, thresholdPercent);\n\n      if (readOnlyIndicesCount > 0) {\n        LOG.warn(\"Elasticsearch indices are in read-only mode, likely due to disk space issues\");\n        serverMonitoringMetrics.setElasticSearchStatusToRed();\n        return;\n      }\n\n      final double freePercent = 100.0 - maxDiskUsagePercent;\n\n      if (freePercent < thresholdPercent) {\n        LOG.warn(\"Elasticsearch nodes have critically low disk space\");\n        serverMonitoringMetrics.setElasticSearchStatusToRed();\n        return;\n      }\n\n      // Fall back to cluster health status\n      if (esStatus == null) {\n        serverMonitoringMetrics.setElasticSearchStatusToRed();\n      } else {\n        switch (esStatus) {\n          case Green, Yellow:\n            serverMonitoringMetrics.setElasticSearchStatusToGreen();\n            break;\n          case Red:\n            serverMonitoringMetrics.setElasticSearchStatusToRed();\n            break;\n        }\n      }\n    } catch (Exception e) {","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/SonarSource/sonarqube/blob/184c821202192afc1c599fc912d0889b69fffa53/server/sonar-webserver-monitoring/src/main/java/org/sonar/server/monitoring/ElasticSearchMetricTask.java#L66-L102","documentation":"The same monitoring task also checks aggregate disk usage across ES nodes. When the percentage of free disk space falls below the configured threshold (default 10%), it logs this warning and sets ES status to Red, warning that indexing is about to fail as nodes approach the flood-stage watermark.","triggerScenarios":"updateElasticSearchHealthStatus (from run) computing freePercent = 100 - maxDiskUsagePercent and finding freePercent < thresholdPercent from property sonar.serverMonitoring.diskSpaceThreshold (default DEFAULT_DISK_SPACE_THRESHOLD_PERCENT).","commonSituations":"Gradual disk growth from analyses/logs; small ES volumes in containers; threshold set higher than actual free space after environment changes; burst indexing during large project imports.","solutions":["Free disk space on the ES node(s) — prune old indices, logs, temp files, or enlarge the volume.","Adjust sonar.serverMonitoring.diskSpaceThreshold if the default (10%) doesn't fit your disk sizing, in sonar.properties.","Set up external disk alerts so you act before the watermark triggers read-only blocks.","Verify with node stats (/_nodes/stats/fs) which node is running out of space."],"exampleFix":"// before: default threshold triggering on a small disk\n# sonar.properties\n// after: tune threshold to 5% for a larger monitored disk\nsonar.serverMonitoring.diskSpaceThreshold=5","handlingStrategy":"validation","validationCode":"const stats = await (await fetch('http://es-host:9001/_nodes/stats/fs')).json();\nfor (const n of Object.values(stats.nodes)) {\n  const freePct = 100 * n.fs.total.free_in_bytes / n.fs.total.total_in_bytes;\n  if (freePct < 10) console.error(`Node ${n.name}: only ${freePct.toFixed(1)}% free disk`);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set up external disk-usage alerts on ES data nodes well above the threshold","Size ES volumes for growth; expand before hitting 90% usage","Tune sonar.serverMonitoring.diskSpaceThreshold to match your capacity strategy","Schedule cleanup of logs and old data instead of reactive deletes"],"tags":["elasticsearch","disk-space","monitoring","capacity"],"backgroundTag":"disk-space-exhausted","analyzedSha":"184c821202192afc1c599fc912d0889b69fffa53","analyzedAt":"2026-09-09T12:23:51.573Z","contentChangedAt":"2026-09-09T12:23:51.573Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}