Crosstalk-Solutions/project-nomad · error · Error

Sysbench command failed: ${error.message}

Error message

Sysbench command failed: ${error.message}

What it means

Generic wrapper thrown by BenchmarkService._runSysbenchCommand when the dockerode container exec/run of sysbench rejects for any reason (image pull failure, container start failure, docker daemon unreachable, exec error). The underlying error is logged with logger.error before rethrowing, so the original cause appears in logs but not in the message chain.

Source

Thrown at admin/app/services/benchmark_service.ts:1621

      try {
        await container.remove()
      } catch (removeError: any) {
        // Log but don't fail if removal fails (container might already be gone)
        logger.warn(`Failed to remove sysbench container: ${removeError.message}`)
      }

      return output
    } catch (error: any) {
      // Clean up container on error if it exists
      if (container) {
        try {
          await container.remove({ force: true })
        } catch (removeError: any) {
          // Ignore removal errors
        }
      }
      logger.error(`Sysbench command failed: ${error.message}`)
      throw new Error(`Sysbench command failed: ${error.message}`)
    }
  }

  /**
   * Like _runSysbenchCommand but attaches to the container's output stream and
   * invokes onLine for every complete line as it arrives, so interim
   * --report-interval lines can drive live telemetry. Still returns the full
   * accumulated stdout so the caller's final-report regexes (the SCORED numbers)
   * parse exactly as before. onLine parsing must never throw.
   */
  // eslint-disable-next-line @typescript-eslint/no-unused-vars
  // @ts-ignore TS6133: Reserved for future live sysbench telemetry mode.
  private async _runSysbenchCommandStreaming(
    cmd: string[],
    onLine: (line: string) => void
  ): Promise<string> {
    let container: Dockerode.Container | null = null
    try {

View on GitHub (pinned to 0bd1c6f4f9)

Solutions

1. Check the server logs for the preceding 'Sysbench command failed: <cause>' line to identify the real error
2. Verify Docker is reachable from the app (docker info / dockerode connection) and the socket is mounted with adequate permissions
3. Pre-pull the sysbench image (docker pull <sysbench image>) on the host or allow outbound registry access
4. Ensure the force-remove cleanup path runs so stale containers don't block subsequent runs
5. Retry the benchmark once after clearing the condition; if the daemon is down, restart it

Example fix

// before
logger.error(`Sysbench command failed: ${error.message}`)
throw new Error(`Sysbench command failed: ${error.message}`)

// after
logger.error(`Sysbench command failed: ${error.message}`)
throw new Error(`Sysbench command failed: ${error.message}`, { cause: error })

const docker = new Docker()
try { await docker.ping() } catch { throw new Error('Docker daemon unreachable — benchmark cannot run') }

try { await svc.runSysbench(...) } catch (e: any) { if (/Sysbench command failed/.test(e.message)) { await backoffRetry(2) } else throw e }

When it happens

Trigger: Calling any benchmark stage that shells out to sysbench via Docker while the Docker socket is unavailable, the sysbench image tag cannot be pulled, the container exits abnormally, or dockerode's start/exec rejects (permissions, daemon down, name conflicts).

Common situations: Docker daemon not running or the admin app lacks access to /var/run/docker.sock, rate-limited or offline image registry pulls, leftover containers with the same name hitting a conflict, or a host with no nested-container support.

Related errors

• sysbench disk-write benchmark produced no parseable MiB/s —
• Warning during container cleanup. Check server logs for deta
• Failed to get auth token from ${registry}: ${response.status
• No token returned from ${registry}
• recreated container ${readiness.reason}

AI-assisted analysis of Crosstalk-Solutions/project-nomad@0bd1c6f4f9 (2026-08-27). Data as JSON: /api/errors/10f14b545004ee32. Report an issue: GitHub.