Crosstalk-Solutions/project-nomad · critical · Error

recreated container ${readiness.reason}

Error message

recreated container ${readiness.reason}

What it means

Thrown by DockerService after recreating a container when the new container fails a post-start readiness/health gate. The service starts a replacement container, waits for it to become healthy via _awaitContainerReady, and if readiness.ready is false it aborts the swap and throws, leaving the old container in place (the removal of the old container only happens after the gate passes).

Source

Thrown at admin/app/services/docker_service.ts:2231

        Labels: {
          ...(containerConfig?.Labels ?? {}),
          'com.docker.compose.project': 'project-nomad-managed',
          'io.project-nomad.managed': 'true',
        },
        ...(containerConfig?.User && { User: containerConfig.User }),
        HostConfig: containerConfig?.HostConfig ?? {},
        ...(containerConfig?.ExposedPorts && { ExposedPorts: containerConfig.ExposedPorts }),
        ...(recreateEnv.length ? { Env: recreateEnv } : {}),
        ...(service.container_command ? { Cmd: service.container_command.split(' ') } : {}),
        ...(process.env.NODE_ENV === 'production' && {
          NetworkingConfig: { EndpointsConfig: { [DockerService.NOMAD_NETWORK]: {} } },
        }),
      })
      await newContainer.start()

      // Health gate before discarding the old container.
      const readiness = await this._awaitContainerReady(newContainer)
      if (!readiness.ready) throw new Error(`recreated container ${readiness.reason}`)

      if (oldInfo) {
        const oldRef = await this._findContainerByName(oldName)
        if (oldRef) await this.docker.getContainer(oldRef.Id).remove({ force: true })
      }
      service.installed = true
      service.installation_status = 'idle'
      await service.save()
      this.invalidateServicesStatusCache()
      return { success: true, message: `Service ${serviceName} reconfigured successfully` }
    } catch (error: any) {
      logger.error({ err: error }, `[DockerService] recreateCustomAppContainer failed for ${serviceName}`)
      // Roll back: discard the failed new container and restore the renamed original.
      try {
        const failedNew = await this._findContainerByName(serviceName)
        if (failedNew) {
          const c = this.docker.getContainer(failedNew.Id)
          await c.stop({ t: 5 }).catch(() => {})

View on GitHub (pinned to 0bd1c6f4f9)

Solutions

1. Inspect readiness.reason in the thrown message to identify whether it's a timeout, unhealthy status, or connection refusal, then check `docker logs <new-container>` for the startup failure
2. Increase the readiness/healthcheck timeout or fix the app's healthcheck definition if the service is just slow to start
3. Verify the new image version is compatible with existing config/volumes (required env vars, DB migrations)
4. Check for port conflicts or leftover containers binding the same port
5. If the old container still runs, the system is safe; fix the root cause and re-trigger the recreate

Example fix

// before
await dockerService.recreateContainer('tipi') // throws: recreated container unhealthy

// after
try {
  await dockerService.recreateContainer('tipi')
} catch (e) {
  // old container is still running; diagnose the new one
  console.error((e as Error).message) // 'recreated container <reason>'
}

// Pre-check image healthcheck definition before recreate
const info = await docker.getContainer(name).inspect()
if (!info.Config.Healthcheck) console.warn('No healthcheck defined; readiness gate may rely on port probe')

const isReadinessError = (e: unknown) =>
  (e as Error).message.startsWith('recreated container')

try {
  await dockerService.recreateContainer(serviceId)
} catch (e) {
  if ((e as Error).message.startsWith('recreated container')) {
    // old container still serves traffic — log reason, alert, retry later
    logger.warn('Recreate failed, old container retained:', (e as Error).message)
    return
  }
  throw e
}

When it happens

Trigger: Calling the container recreate/update flow (e.g. app update or reinstall) where the new container starts but its health check fails, times out, or the app inside never becomes reachable; readiness.reason carries the underlying cause (health status, timeout, port not listening).

Common situations: Updated image that crashes on boot, missing env vars/migrations for the new version, healthcheck misconfigured or too short a timeout, port conflicts preventing the new container from binding, slow image startup exceeding the readiness wait window.

Related errors

• sysbench disk-write benchmark produced no parseable MiB/s —
• Sysbench command failed: ${error.message}
• Failed to get auth token from ${registry}: ${response.status
• No token returned from ${registry}
• Failed to fetch available versions for this service.

AI-assisted analysis of Crosstalk-Solutions/project-nomad@0bd1c6f4f9 (2026-08-27). Data as JSON: /api/errors/36c1ff84ab446457. Report an issue: GitHub.