{"record":{"id":"08010a96f563fec9","repo":"SonarSource/sonarqube","slug":"index-is-in-read-only-mode-index-blocks-read","errorCode":null,"errorMessage":"Index [{}] is in read-only mode (index.blocks.read_only_allow_delete=true)","messagePattern":"Index \\[(.+?)\\] is in read-only mode \\(index\\.blocks\\.read_only_allow_delete=true\\)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/sonar-webserver-monitoring/src/main/java/org/sonar/server/monitoring/ElasticSearchMetricTask.java","lineNumber":136,"sourceCode":"    } catch (Exception e) {\n      LOG.error(\"Failed to query ES status\", e);\n    }\n  }\n\n  private long updateAndGetReadOnlyIndicesCount() {\n    try {\n      final GetIndicesSettingsResponse settingsResponse = esClient.getSettingsV2(req -> req.index(\"*\"));\n      final Map<String, IndexState> indices = settingsResponse.settings();\n\n      long readOnlyCount = 0;\n      for (Map.Entry<String, IndexState> entry : indices.entrySet()) {\n        final String indexName = entry.getKey();\n        final IndexState indexState = entry.getValue();\n\n        if (indexState.settings() != null && indexState.settings().index() != null) {\n          final var blocks = indexState.settings().index().blocks();\n          if (blocks != null && Boolean.TRUE.equals(blocks.readOnlyAllowDelete())) {\n            LOG.warn(\"Index [{}] is in read-only mode (index.blocks.read_only_allow_delete=true)\", indexName);\n            readOnlyCount++;\n          }\n        }\n      }\n\n      // Update metric for observability\n      serverMonitoringMetrics.setElasticSearchReadOnlyIndicesCount(readOnlyCount);\n\n      return readOnlyCount;\n    } catch (Exception e) {\n      LOG.error(\"Failed to check for readonly indices\", e);\n      // Return 0 on error to avoid false positives - let cluster health handle it\n      return 0;\n    }\n  }\n\n  private double updateAndGetMaxDiskUsagePercent(@Nullable NodeStatsResponse nodeStatsResponse, double thresholdPercent) {\n    try {","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/SonarSource/sonarqube/blob/184c821202192afc1c599fc912d0889b69fffa53/server/sonar-webserver-monitoring/src/main/java/org/sonar/server/monitoring/ElasticSearchMetricTask.java#L118-L154","documentation":"When scanning per-index settings returned by the ES stats API, the task counts indices whose index.blocks.read_only_allow_delete block is true and logs one warning per blocked index. This block is typically set by Elasticsearch's flood-stage watermark to protect the disk, making the index read-only (deletes allowed).","triggerScenarios":"updateAndGetReadOnlyIndicesCount (called from readOnlyIndicesCount path) iterating index states where settings().index().blocks().readOnlyAllowDelete() equals TRUE for a given index name.","commonSituations":"Disk crossed 95% on the data node causing ES to auto-block indices; indices left blocked after disk cleanup because the block is not removed automatically (pre-7.x behavior / manual setting); restored snapshots carrying block settings.","solutions":["Free disk space above the flood-stage watermark on the ES data node.","Remove the block per index: PUT /<indexName>/_settings {\"index.blocks.read_only_allow_delete\": null}.","Check cluster settings for any manually applied read_only_allow_delete blocks and clear them.","Confirm via GET /_all/_settings that no blocks remain, then verify ES status turns Green in SonarQube monitoring."],"exampleFix":"# before: index blocked\ncurl -XGET 'localhost:9001/sonarqube/_settings?pretty' # \"read_only_allow_delete\": \"true\"\n# after: unblock once disk is freed\ncurl -XPUT 'localhost:9001/sonarqube/_settings' -H 'Content-Type: application/json' -d '{\"index.blocks.read_only_allow_delete\": null}'","handlingStrategy":"retry","validationCode":"# verify no blocked indices before indexing\ncurl -s 'localhost:9001/_all/_settings' | jq '[to_entries[] | select(.value.settings.index.blocks.read_only_allow_delete==\"true\") | .key]'","typeGuard":null,"tryCatchPattern":"const blocked = await getBlockedIndices();\nif (blocked.length > 0) {\n  await freeDiskSpaceThenRetry(async () => {\n    for (const idx of blocked) await es.indices.putSettings({ index: idx, body: { 'index.blocks.read_only_allow_delete': null } });\n  });\n}","preventionTips":["Keep ES disk usage below the flood-stage watermark (95%) at all times","After freeing space, explicitly remove the per-index block — it doesn't clear itself","Check for restored snapshots or copied data dirs carrying block settings","Include a blocked-index check in routine ES health scripts"],"tags":["elasticsearch","read-only","index-block","disk-space"],"backgroundTag":"disk-space-exhausted","analyzedSha":"184c821202192afc1c599fc912d0889b69fffa53","analyzedAt":"2026-09-09T12:23:51.573Z","contentChangedAt":"2026-09-09T12:23:51.573Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}