SonarSource/sonarqube · error

Elasticsearch nodes have critically low disk space

Error message

Elasticsearch nodes have critically low disk space

What it means

The same monitoring task also checks aggregate disk usage across ES nodes. When the percentage of free disk space falls below the configured threshold (default 10%), it logs this warning and sets ES status to Red, warning that indexing is about to fail as nodes approach the flood-stage watermark.

Solutions

1. Free disk space on the ES node(s) — prune old indices, logs, temp files, or enlarge the volume.
2. Adjust sonar.serverMonitoring.diskSpaceThreshold if the default (10%) doesn't fit your disk sizing, in sonar.properties.
3. Set up external disk alerts so you act before the watermark triggers read-only blocks.
4. Verify with node stats (/_nodes/stats/fs) which node is running out of space.

Example fix

// before: default threshold triggering on a small disk
# sonar.properties
// after: tune threshold to 5% for a larger monitored disk
sonar.serverMonitoring.diskSpaceThreshold=5

const stats = await (await fetch('http://es-host:9001/_nodes/stats/fs')).json();
for (const n of Object.values(stats.nodes)) {
  const freePct = 100 * n.fs.total.free_in_bytes / n.fs.total.total_in_bytes;
  if (freePct < 10) console.error(`Node ${n.name}: only ${freePct.toFixed(1)}% free disk`);
}

When it happens

Trigger: updateElasticSearchHealthStatus (from run) computing freePercent = 100 - maxDiskUsagePercent and finding freePercent < thresholdPercent from property sonar.serverMonitoring.diskSpaceThreshold (default DEFAULT_DISK_SPACE_THRESHOLD_PERCENT).

Common situations: Gradual disk growth from analyses/logs; small ES volumes in containers; threshold set higher than actual free space after environment changes; burst indexing during large project imports.

Related errors

• Elasticsearch indices are in read-only mode, likely due to…
• Node [ ] has critically low disk space: % free ( bytes…
• Failed to retrieve ES attributes. There will be only a…
• Index [ ] is in read-only mode…
• At least one Elasticsearch host is required

AI-assisted analysis of SonarSource/sonarqube@184c821202 (2026-09-09). Data as JSON: /api/errors/aab466e9fd589db1. Report an issue: GitHub.

Appendix: source

Thrown at server/sonar-webserver-monitoring/src/main/java/org/sonar/server/monitoring/ElasticSearchMetricTask.java:84

  private void updateElasticSearchHealthStatus(@Nullable final NodeStatsResponse nodeStatsResponse) {
    try {
      HealthStatus esStatus = esClient.clusterHealthV2(req -> req).status();

      long readOnlyIndicesCount = updateAndGetReadOnlyIndicesCount();
      final double thresholdPercent = config.getDouble(DISK_SPACE_THRESHOLD_PROPERTY)
        .orElse(DEFAULT_DISK_SPACE_THRESHOLD_PERCENT);
      double maxDiskUsagePercent = updateAndGetMaxDiskUsagePercent(nodeStatsResponse, thresholdPercent);

      if (readOnlyIndicesCount > 0) {
        LOG.warn("Elasticsearch indices are in read-only mode, likely due to disk space issues");
        serverMonitoringMetrics.setElasticSearchStatusToRed();
        return;
      }

      final double freePercent = 100.0 - maxDiskUsagePercent;

      if (freePercent < thresholdPercent) {
        LOG.warn("Elasticsearch nodes have critically low disk space");
        serverMonitoringMetrics.setElasticSearchStatusToRed();
        return;
      }

      // Fall back to cluster health status
      if (esStatus == null) {
        serverMonitoringMetrics.setElasticSearchStatusToRed();
      } else {
        switch (esStatus) {
          case Green, Yellow:
            serverMonitoringMetrics.setElasticSearchStatusToGreen();
            break;
          case Red:
            serverMonitoringMetrics.setElasticSearchStatusToRed();
            break;
        }
      }
    } catch (Exception e) {

View on GitHub (pinned to 184c821202)