{"record":{"id":"0cb3d00ff03991e2","repo":"coollabsio/coolify","slug":"something-is-not-okay-are-you-okay","errorCode":null,"errorMessage":"Something is not okay, are you okay?","messagePattern":"Something is not okay, are you okay\\?","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"app/Traits/HasMetrics.php","lineNumber":63,"sourceCode":"        }\n        $token = $server->settings->ensureValidSentinelToken();\n        if ($token !== $previousToken) {\n            Log::warning('Regenerated sentinel token during metrics read; sentinel container restart required', ['server_id' => $server->id]);\n        }\n\n        $response = instant_remote_process(\n            [\"docker exec coolify-sentinel sh -c 'curl -H \\\"Authorization: Bearer {$token}\\\" {$endpoint}'\"],\n            $server,\n            false\n        );\n\n        if (str($response)->contains('error')) {\n            $error = json_decode($response, true);\n            $error = data_get($error, 'error', 'Something is not okay, are you okay?');\n            if ($error === 'Unauthorized') {\n                $error = 'Unauthorized, please check your metrics token or restart Sentinel to set a new token.';\n            }\n            throw new \\Exception($error);\n        }\n\n        $metrics = collect(json_decode($response, true))->map(function ($metric) use ($valueField) {\n            return [(int) $metric['time'], (float) ($metric[$valueField] ?? 0.0)];\n        })->toArray();\n\n        if ($mins > 60 && count($metrics) > 1000) {\n            $metrics = downsampleLTTB($metrics, 1000);\n        }\n\n        return $metrics;\n    }\n\n    private function isServerMetrics(): bool\n    {\n        return $this instanceof Server;\n    }\n","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/coollabsio/coolify/blob/70b9acc42467278373e00de77abb40684e25b395/app/Traits/HasMetrics.php#L45-L81","documentation":"Coolify's HasMetrics trait fetches CPU/memory metrics by shelling into the server (instant_remote_process), docker-exec'ing into the coolify-sentinel container, and curling its HTTP API (http://localhost:8888/api/...) with a bearer token. When the returned string contains 'error' anywhere, the trait tries to json_decode it and read the 'error' key; if the payload is not valid JSON (or has no 'error' key), data_get falls back to the generic message 'Something is not okay, are you okay?'. So this exception means: the sentinel metrics call came back with something error-shaped, but the actual cause was lost because the response could not be parsed.","triggerScenarios":"Calling getCpuMetrics()/getMemoryMetrics() on a Server or container-backed model when: (1) the sentinel API replies {\"error\":\"Unauthorized\"} because the bearer token from server settings no longer matches the token the running coolify-sentinel container holds (token was regenerated by ensureValidSentinelToken, e.g. after a DecryptException on the stored sentinel_token); (2) the curl/docker exec output merely contains the substring 'error' (container stderr, docker daemon text, partial/HTML output) but is not JSON, so json_decode returns null; (3) the coolify-sentinel container is stopped or missing so the docker exec itself errors; (4) the requested metrics history (e.g. /container/{uuid}/cpu/history?from=...) returns an error payload for an unknown/unscraped container.","commonSituations":"Seen after Coolify upgrades that change token encryption/format (old sentinel_token fails DecryptException, a fresh token is minted server-side but the still-running sentinel container keeps the old one); on freshly added servers where sentinel has not started or has not collected data yet; when server metrics are enabled in settings but the sentinel container is unhealthy (disk full, crashed); or when anything non-JSON leaks into the curl response and trips the substring check.","solutions":["Restart the sentinel container on the affected server (docker restart coolify-sentinel) so it picks up the newly regenerated metrics token — this fixes the most common cause, token mismatch after regeneration.","Verify the container exists and is healthy: docker ps --filter name=coolify-sentinel and docker logs coolify-sentinel (look for startup/token errors).","Reproduce the raw response to see the real error: docker exec coolify-sentinel sh -c 'curl -H \"Authorization: Bearer <token>\" http://localhost:8888/api/cpu/history?from=<iso-date>' using the token from the server's settings.","Confirm metrics are enabled for the server (Server Settings -> Metrics); if disabled the trait returns null instead of fetching.","If the stored token is undecryptable (DecryptException keeps firing), clear sentinel_token in server settings so ensureValidSentinelToken issues a fresh one, then restart sentinel."],"exampleFix":"// before\n$metrics = $application->getMemoryMetrics($mins); // throws when sentinel errors or response is unparsable\n\n// after\nuse Illuminate\\Support\\Facades\\Log;\n\ntry {\n    $metrics = $application->getMemoryMetrics($mins);\n} catch (\\Exception $e) {\n    Log::warning('Metrics fetch failed, showing empty chart', [\n        'error' => $e->getMessage(),\n        'server' => $application->destination->server->id,\n    ]);\n    $metrics = null;\n}","handlingStrategy":"fallback","validationCode":"$server = $application->destination->server;\nif (! $server->isMetricsEnabled()) {\n    return null; // metrics disabled; the trait would return null anyway\n}\n// Optional pre-flight: confirm sentinel is actually up before fetching\n$status = instant_remote_process(\n    [\"docker inspect -f '{{.State.Status}}' coolify-sentinel\"],\n    $server,\n    false\n);\nif (trim((string) $status) !== 'running') {\n    return null; // avoid guaranteed-to-fail fetch\n}","typeGuard":null,"tryCatchPattern":"use Illuminate\\Support\\Facades\\Log;\n\ntry {\n    $metrics = $server->getCpuMetrics($mins);\n} catch (\\Exception $e) {\n    // Metrics are non-critical UI data: degrade to an empty chart, never fail the page/job\n    Log::warning('Sentinel metrics unavailable', ['server' => $server->id, 'error' => $e->getMessage()]);\n    $metrics = null;\n}\n// treat null as 'no data' in the view instead of throwing","preventionTips":["Keep the coolify-sentinel container running and healthy on every managed server; restart it after any token change or Coolify upgrade.","After regenerating a metrics token, always restart sentinel in the same operation so both sides hold the same token.","Gate fetches on $server->isMetricsEnabled() and a sentinel health check instead of calling the metrics API blindly.","Smoke-test the sentinel endpoint with a manual docker exec curl when wiring up new servers.","Alert on sentinel container restarts/exits so token drift is caught before users see failed charts."],"tags":["metrics","sentinel","docker","token","coolify"],"backgroundTag":"upstream-api-error","analyzedSha":"70b9acc42467278373e00de77abb40684e25b395","analyzedAt":"2026-08-17T01:41:01.313Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}