coollabsio/coolify · error · Exception

Something is not okay, are you okay?

Error message

Something is not okay, are you okay?

What it means

Coolify's HasMetrics trait fetches CPU/memory metrics by shelling into the server (instant_remote_process), docker-exec'ing into the coolify-sentinel container, and curling its HTTP API (http://localhost:8888/api/...) with a bearer token. When the returned string contains 'error' anywhere, the trait tries to json_decode it and read the 'error' key; if the payload is not valid JSON (or has no 'error' key), data_get falls back to the generic message 'Something is not okay, are you okay?'. So this exception means: the sentinel metrics call came back with something error-shaped, but the actual cause was lost because the response could not be parsed.

Source

Thrown at app/Traits/HasMetrics.php:63

        }
        $token = $server->settings->ensureValidSentinelToken();
        if ($token !== $previousToken) {
            Log::warning('Regenerated sentinel token during metrics read; sentinel container restart required', ['server_id' => $server->id]);
        }

        $response = instant_remote_process(
            ["docker exec coolify-sentinel sh -c 'curl -H \"Authorization: Bearer {$token}\" {$endpoint}'"],
            $server,
            false
        );

        if (str($response)->contains('error')) {
            $error = json_decode($response, true);
            $error = data_get($error, 'error', 'Something is not okay, are you okay?');
            if ($error === 'Unauthorized') {
                $error = 'Unauthorized, please check your metrics token or restart Sentinel to set a new token.';
            }
            throw new \Exception($error);
        }

        $metrics = collect(json_decode($response, true))->map(function ($metric) use ($valueField) {
            return [(int) $metric['time'], (float) ($metric[$valueField] ?? 0.0)];
        })->toArray();

        if ($mins > 60 && count($metrics) > 1000) {
            $metrics = downsampleLTTB($metrics, 1000);
        }

        return $metrics;
    }

    private function isServerMetrics(): bool
    {
        return $this instanceof Server;
    }

View on GitHub (pinned to 70b9acc424)

Solutions

  1. Restart the sentinel container on the affected server (docker restart coolify-sentinel) so it picks up the newly regenerated metrics token — this fixes the most common cause, token mismatch after regeneration.
  2. Verify the container exists and is healthy: docker ps --filter name=coolify-sentinel and docker logs coolify-sentinel (look for startup/token errors).
  3. Reproduce the raw response to see the real error: docker exec coolify-sentinel sh -c 'curl -H "Authorization: Bearer <token>" http://localhost:8888/api/cpu/history?from=<iso-date>' using the token from the server's settings.
  4. Confirm metrics are enabled for the server (Server Settings -> Metrics); if disabled the trait returns null instead of fetching.
  5. If the stored token is undecryptable (DecryptException keeps firing), clear sentinel_token in server settings so ensureValidSentinelToken issues a fresh one, then restart sentinel.

Example fix

// before
$metrics = $application->getMemoryMetrics($mins); // throws when sentinel errors or response is unparsable

// after
use Illuminate\Support\Facades\Log;

try {
    $metrics = $application->getMemoryMetrics($mins);
} catch (\Exception $e) {
    Log::warning('Metrics fetch failed, showing empty chart', [
        'error' => $e->getMessage(),
        'server' => $application->destination->server->id,
    ]);
    $metrics = null;
}
Defensive patterns

Strategy: fallback

Validate before calling

$server = $application->destination->server;
if (! $server->isMetricsEnabled()) {
    return null; // metrics disabled; the trait would return null anyway
}
// Optional pre-flight: confirm sentinel is actually up before fetching
$status = instant_remote_process(
    ["docker inspect -f '{{.State.Status}}' coolify-sentinel"],
    $server,
    false
);
if (trim((string) $status) !== 'running') {
    return null; // avoid guaranteed-to-fail fetch
}

Try / catch

use Illuminate\Support\Facades\Log;

try {
    $metrics = $server->getCpuMetrics($mins);
} catch (\Exception $e) {
    // Metrics are non-critical UI data: degrade to an empty chart, never fail the page/job
    Log::warning('Sentinel metrics unavailable', ['server' => $server->id, 'error' => $e->getMessage()]);
    $metrics = null;
}
// treat null as 'no data' in the view instead of throwing

Prevention

When it happens

Trigger: Calling getCpuMetrics()/getMemoryMetrics() on a Server or container-backed model when: (1) the sentinel API replies {"error":"Unauthorized"} because the bearer token from server settings no longer matches the token the running coolify-sentinel container holds (token was regenerated by ensureValidSentinelToken, e.g. after a DecryptException on the stored sentinel_token); (2) the curl/docker exec output merely contains the substring 'error' (container stderr, docker daemon text, partial/HTML output) but is not JSON, so json_decode returns null; (3) the coolify-sentinel container is stopped or missing so the docker exec itself errors; (4) the requested metrics history (e.g. /container/{uuid}/cpu/history?from=...) returns an error payload for an unknown/unscraped container.

Common situations: Seen after Coolify upgrades that change token encryption/format (old sentinel_token fails DecryptException, a fresh token is minted server-side but the still-running sentinel container keeps the old one); on freshly added servers where sentinel has not started or has not collected data yet; when server metrics are enabled in settings but the sentinel container is unhealthy (disk full, crashed); or when anything non-JSON leaks into the curl response and trips the substring check.

Related errors

AI-assisted analysis of coollabsio/coolify@70b9acc424 (2026-08-17). Data as JSON: /api/errors/0cb3d00ff03991e2. Report an issue: GitHub.