microsoft/garnet · error

Failed to schedule work: {}

Error message

Failed to schedule work: {}

What it means

Returned by WindowsPtpThreadPool::Schedule when CreateThreadpoolWork returns NULL. The pool allocates a TaskInfo, tries to register a work object against the Win32 thread pool bound in the constructor; on failure it formats FormatWin32AndHRESULT(GetLastError()), prints to stderr, and returns Status::Aborted (note: Aborted, not IOError). The TaskInfo is freed by alloc_context's deleter; no work runs.

Source

Thrown at libs/storage/Tsavorite/cc/src/device/file_windows.cc:183

  ::CloseThreadpool(pool_);

  delete callback_environment_;
}

Status WindowsPtpThreadPool::Schedule(Task task, void* task_parameters) {
  auto info = alloc_context<TaskInfo>(sizeof(TaskInfo));
  if(!info.get()) return Status::OutOfMemory;
  new(info.get()) TaskInfo();

  info->task = task;
  info->task_parameters = task_parameters;

  PTP_WORK_CALLBACK ptp_callback = TaskStartSpringboard;
  PTP_WORK work = CreateThreadpoolWork(ptp_callback, info.get(), callback_environment_);
  if(!work) {
    std::stringstream ss;
    ss << "Failed to schedule work: " << FormatWin32AndHRESULT(::GetLastError());
    fprintf(stderr, "%s\n", ss.str().c_str());
    return Status::Aborted;
  }
  SubmitThreadpoolWork(work);
  info.release();

  return Status::Ok;
}

void CALLBACK WindowsPtpThreadPool::TaskStartSpringboard(PTP_CALLBACK_INSTANCE instance,
    PVOID parameter, PTP_WORK work) {
  auto info = make_context_unique_ptr<TaskInfo>(reinterpret_cast<TaskInfo*>(parameter));
  info->task(info->task_parameters);
  CloseThreadpoolWork(work);
}

Status ThreadPoolFile::Open(FileCreateDisposition create_disposition, const FileOptions& options,
                            ThreadPoolIoHandler* handler, bool* exists) {
  DWORD flags = FILE_FLAG_RANDOM_ACCESS | FILE_FLAG_OVERLAPPED;

View on GitHub (pinned to 951b0fc683)

Solutions

1. Stop all callers of Schedule (quiesce pending async ops, e.g. the device's numPending) before destroying the thread pool / closing cleanup group members.
2. Check the Status return from Schedule (and from ReadAsync/WriteAsync which surface it) and treat Status::Aborted as fatal-to-operation rather than continuing.
3. Inspect the HRESULT in the stderr line — ERROR_INVALID_PARAMETER points to environment teardown ordering; resource codes point to quota/memory.
4. If genuinely transient (brief resource pressure), retry Schedule a bounded number of times with backoff, otherwise propagate the failure to the caller.

Example fix

// before:
// Status s = pool.Schedule(task, ctx);
// after:
Status s = pool.Schedule(task, ctx);
if (s != Status::Ok) {
  // Aborted => threadpool torn down or out of resources; do not retry indefinitely
  FinishRequest(ctx, s);
  return s;
}

// Validate the thread pool is live before submitting work.
if (pool == nullptr || pool->IsClosed()) return Status::Aborted;
// Then check the returned Status from Schedule; Aborted means do not silently continue.

When it happens

Trigger: CreateThreadpoolWork fails: the thread pool or its cleanup group has already been closed/destroyed (e.g. Schedule called after the WindowsPtpThreadPool destructor began closing members), the callback_environment_ is invalid, or the system is out of resources (memory / handles / thread quota). GetLastError typically yields ERROR_INVALID_PARAMETER (environment torn down) or a resource-exhaustion code.

Common situations: Submitting tasks during or after device/thread-pool shutdown; a destructor racing with in-flight Schedule calls on another thread; running under a low handle/quota job object; very high async checkpoint/grow concurrency exhausting the pool; reusing a FASTER instance whose thread pool was already disposed.

Related errors

• Failed to schedule async IO: {}
• Failed to schedule async IO: {}, handle {}
• Slowdown: Unable to add trigger to epoch
• MutablePercent must be between 10 and 95
• Store Log Memory size or PageCount must be specified

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/397e8e1b896ee27d. Report an issue: GitHub.