gastownhall/beads · warning

server: DoltServer.Stop: remove pidfile: %w

Error message

server: DoltServer.Stop: remove pidfile: %w

What it means

DoltServer.Stop aggregates all shutdown errors via errors.Join, and this one wraps any failure from pidfile.Remove, which deletes the server's PID file from the root directory after the process has stopped. It means the server process itself shut down, but the bookkeeping file recording its PID could not be removed from disk.

Source

Thrown at internal/storage/dbproxy/server/doltserver.go:382

	}
	if waitErr != nil {
		waitErr = fmt.Errorf("server: DoltServer.Stop: %w", waitErr)
	}
	var closeErr error
	if s.logFile != nil {
		closeErr = s.logFile.Close()
		s.logFile = nil
	}
	if closeErr != nil {
		closeErr = fmt.Errorf("server: DoltServer.Stop: close log: %w", closeErr)
	}
	var rmErr error
	if s.pid != 0 {
		rmErr = pidfile.Remove(s.rootDir, PIDFileName)
		s.pid = 0
	}
	if rmErr != nil {
		rmErr = fmt.Errorf("server: DoltServer.Stop: remove pidfile: %w", rmErr)
	}
	return errors.Join(gcErr, waitErr, closeErr, rmErr)
}

func (s *DoltServer) runShutdownGC(ctx context.Context) (retErr error) {
	if s.database == "" || !s.Running(ctx) {
		return nil
	}
	db, err := sql.Open("mysql", s.DSN(ctx, s.database, "root", ""))
	if err != nil {
		return fmt.Errorf("open gc connection: %w", err)
	}
	defer func() { retErr = errors.Join(retErr, db.Close()) }()

	conn, err := db.Conn(ctx)
	if err != nil {
		return fmt.Errorf("acquire gc connection: %w", err)
	}

View on GitHub (pinned to 71377f2769)

Solutions

1. Check permissions/ownership of the server root directory and the PID file; ensure the stopping process can write there
2. Verify the root directory still exists and is writable before calling Stop
3. If the error is benign (stale pidfile in a tmpfs), remove the PID file manually and retry Stop
4. Do not treat this as a server failure: GC/wait/close errors in the joined error are the ones that matter for data integrity

Example fix

// before
if err := server.Stop(ctx); err != nil { return err }
// after
if err := server.Stop(ctx); err != nil {
    if !strings.Contains(err.Error(), "remove pidfile") {
        return err // real shutdown problem
    }
    log.Warn("server stopped but pidfile removal failed; cleaning up")
    os.Remove(filepath.Join(rootDir, PIDFileName))
}

// before Stop
if info, err := os.Stat(rootDir); err != nil || !info.IsDir() {
    return fmt.Errorf("server root %s missing or not a directory", rootDir)
}
if err := unix.Access(rootDir, unix.W_OK); err != nil {
    return fmt.Errorf("server root %s not writable: %w", rootDir, err)
}

func isPidfileErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "remove pidfile")
}

if err := server.Stop(ctx); err != nil {
    if isPidfileErr(err) {
        log.Warn("non-fatal pidfile cleanup failure", "err", err)
    } else {
        return err
    }
}

When it happens

Trigger: Calling DoltServer.Stop (via stopWithTimeout) when s.pid != 0 and pidfile.Remove fails to unlink <rootDir>/PIDFileName — e.g. permissions changed on the root dir, the directory was deleted concurrently, or the filesystem is read-only.

Common situations: Running the server as one user and stopping it as another (PID file owned by the original user); root directory mounted read-only at shutdown; container filesystem teardown racing Stop; antivirus/indexers briefly locking the file on network volumes.

Related errors

• read backend record %s: %w
• quarantine dead unverifiable backend record: %w
• quarantine orphan backend record: %w
• proxy.ForceStopUnverified: re-read %s: %w
• write pid file: %w

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/12b4839752d2fec9. Report an issue: GitHub.