{"record":{"id":"43dc5c46bcc131b0","repo":"SonarSource/sonarqube","slug":"elasticsearch-indices-are-in-read-only-mode-likel","errorCode":null,"errorMessage":"Elasticsearch indices are in read-only mode, likely due to disk space issues","messagePattern":"Elasticsearch indices are in read-only mode, likely due to disk space issues","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/sonar-webserver-monitoring/src/main/java/org/sonar/server/monitoring/ElasticSearchMetricTask.java","lineNumber":76,"sourceCode":"      nodeStatsResponse = esClient.nodesStats();\n    } catch (Exception e) {\n      LOG.error(\"Failed to query ES node stats\", e);\n    }\n    updateElasticSearchHealthStatus(nodeStatsResponse);\n    updateFileSystemMetrics(nodeStatsResponse);\n  }\n\n  private void updateElasticSearchHealthStatus(@Nullable final NodeStatsResponse nodeStatsResponse) {\n    try {\n      HealthStatus esStatus = esClient.clusterHealthV2(req -> req).status();\n\n      long readOnlyIndicesCount = updateAndGetReadOnlyIndicesCount();\n      final double thresholdPercent = config.getDouble(DISK_SPACE_THRESHOLD_PROPERTY)\n        .orElse(DEFAULT_DISK_SPACE_THRESHOLD_PERCENT);\n      double maxDiskUsagePercent = updateAndGetMaxDiskUsagePercent(nodeStatsResponse, thresholdPercent);\n\n      if (readOnlyIndicesCount > 0) {\n        LOG.warn(\"Elasticsearch indices are in read-only mode, likely due to disk space issues\");\n        serverMonitoringMetrics.setElasticSearchStatusToRed();\n        return;\n      }\n\n      final double freePercent = 100.0 - maxDiskUsagePercent;\n\n      if (freePercent < thresholdPercent) {\n        LOG.warn(\"Elasticsearch nodes have critically low disk space\");\n        serverMonitoringMetrics.setElasticSearchStatusToRed();\n        return;\n      }\n\n      // Fall back to cluster health status\n      if (esStatus == null) {\n        serverMonitoringMetrics.setElasticSearchStatusToRed();\n      } else {\n        switch (esStatus) {\n          case Green, Yellow:","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/SonarSource/sonarqube/blob/184c821202192afc1c599fc912d0889b69fffa53/server/sonar-webserver-monitoring/src/main/java/org/sonar/server/monitoring/ElasticSearchMetricTask.java#L58-L94","documentation":"The Elasticsearch monitoring task sets the ES health metric. When at least one ES index carries the read_only_allow_delete block (usually applied automatically by ES flood-stage watermark when disk usage exceeds 95%), the task logs this warning and marks the Elasticsearch status Red, signaling that writes (indexing) are failing or will fail.","triggerScenarios":"updateElasticSearchHealthStatus (from run) finding updateAndGetReadOnlyIndicesCount() > 0, i.e. any index reporting index.blocks.read_only_allow_delete=true in node/cluster stats.","commonSituations":"Disk on the ES data node filled past the 95% flood-stage watermark; undersized disk for large analysis histories; Docker/Kubernetes volume limits reached; forgotten watermark settings after expanding storage.","solutions":["Free disk space on the Elasticsearch data node (delete old indices, logs, or expand the volume).","Once free space is above the watermark, remove the block: PUT /<index>/_settings {\"index.blocks.read_only_allow_delete\": null}.","Raise or tune cluster routing.disk.watermark settings if the threshold is inappropriate for your environment.","Restart SonarQube's ES node after cleanup and confirm status returns to Green."],"exampleFix":"# before: index blocked\ncurl -XGET localhost:9001/_all/_settings | grep read_only_allow_delete\n# after: free disk space, then unblock\ncurl -XPUT 'localhost:9001/sonarqube/_settings' -H 'Content-Type: application/json' -d '{\"index.blocks.read_only_allow_delete\": null}'","handlingStrategy":"retry","validationCode":"# before running analyses, check for read-only blocks and disk\nwhile read idx; do echo \"$idx is blocked\"; done < <(curl -s localhost:9001/_all/_settings | jq -r 'to_entries[] | select(.value.settings.index.blocks.read_only_allow_delete==\"true\") | .key')\ncurl -s localhost:9001/_nodes/stats/fs | jq '.nodes[] | {name:.name, free:.fs.total.free_in_bytes, total:.fs.total.total_in_bytes}'","typeGuard":null,"tryCatchPattern":"// poll until the block clears before retrying indexing\nfor (let i = 0; i < 5; i++) {\n  const blocks = await checkReadOnlyBlocks();\n  if (blocks === 0) return retryIndexing();\n  await sleep(60_000);\n}","preventionTips":["Alert when ES disk usage exceeds 85% (before the 95% flood-stage watermark)","Provision disk headroom for peak indexing loads","Automatically clear read_only_allow_delete blocks after freeing space","Monitor SonarQube's ES status metric; treat Red as actionable"],"tags":["elasticsearch","disk-space","read-only","monitoring"],"backgroundTag":"disk-space-exhausted","analyzedSha":"184c821202192afc1c599fc912d0889b69fffa53","analyzedAt":"2026-09-09T12:23:51.573Z","contentChangedAt":"2026-09-09T12:23:51.573Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}