microsoft/garnet · error

Failed to schedule async IO: {}

Error message

Failed to schedule async IO: {}

What it means

Returned by ThreadPoolFile::ScheduleOperation when ReadFile/WriteFile fails with a Win32 code other than ERROR_IO_PENDING. The handler uses the Win32 threadpool I/O object (StartThreadpoolIo was called first); on failure it calls CancelThreadpoolIo to balance the pending count, formats FormatWin32AndHRESULT(win32_result), prints to stderr, and returns Status::IOError. The io_context is freed (not released) so no completion callback fires.

Source

Thrown at libs/storage/Tsavorite/cc/src/device/file_windows.cc:268

      callback);

  ::StartThreadpoolIo(io_object_);

  bool success = FALSE;
  if(FileOperationType::Read == operationType) {
    success = ::ReadFile(file_handle_, buffer, length, nullptr, &io_context->parent_overlapped);
  } else {
    success = ::WriteFile(file_handle_, buffer, length, nullptr, &io_context->parent_overlapped);
  }
  if(!success) {
    DWORD win32_result = ::GetLastError();
    // Any error other than ERROR_IO_PENDING means the IO failed. Otherwise it will finish
    // asynchronously on the threadpool
    if(ERROR_IO_PENDING != win32_result) {
      ::CancelThreadpoolIo(io_object_);
      std::stringstream ss;
      ss << "Failed to schedule async IO: " << FormatWin32AndHRESULT(win32_result);
      fprintf(stderr, "%s\n", ss.str().c_str());
      return Status::IOError;
    }
  }
  io_context.release();
  return Status::Ok;
}

bool QueueIoHandler::TryComplete() {
  DWORD bytes_transferred;
  ULONG_PTR completion_key;
  LPOVERLAPPED overlapped = NULL;
  bool succeeded = ::GetQueuedCompletionStatus(io_completion_port_, &bytes_transferred,
                   &completion_key, &overlapped, 0);
  if(overlapped) {
    Status return_status;
    if(!succeeded) {
      return_status = Status::IOError;
    } else {

View on GitHub (pinned to 951b0fc683)

Solutions

1. Enforce sector alignment of offset, length, and buffer for every Read/Write on unbuffered devices — read device alignment from the file (GetDeviceAlignment) and round up; pin buffers via aligned allocation.
2. Decode the Win32 code in the stderr line: ERROR_INVALID_PARAMETER (87) almost always means misalignment; ERROR_HANDLE_EOF/ERROR_INVALID_HANDLE means handle state; network codes mean transport.
3. Do not issue IO after Close()/device Dispose; gate IO on a valid file_handle_ and a disposed flag.
4. For transient transport errors (network shares), retry the operation with backoff; for alignment errors fix the buffer/size, do not retry unchanged.

Example fix

// before:
// file.Read(offset, length, buffer, ctx, cb);   // length not sector-aligned on unbuffered device
// after (align to device sector size):
size_t sector = file.device_alignment();
assert(offset % sector == 0 && length % sector == 0 &&
       reinterpret_cast<uintptr_t>(buffer) % sector == 0);
file.Read(offset, length, buffer, ctx, cb);

// Validate alignment before issuing overlapped IO on a (possibly unbuffered) device.
size_t sector = file.device_alignment();
if (offset % sector != 0 || length % sector != 0 ||
    reinterpret_cast<uintptr_t>(buffer) % sector != 0)
    return Status::IOError;   // surface misalignment explicitly
if (file.handle() == INVALID_HANDLE_VALUE) return Status::IOError;

When it happens

Trigger: ReadFile/WriteFile fails synchronously on an overlapped handle. Dominant causes: unbuffered I/O (FILE_FLAG_NO_BUFFERING) with offset/length/buffer not aligned to device sector size (the DCHECK_ALIGNMENT assert is only compiled in _DEBUG, so misalignment slips through in release); an invalid/closed file_handle_; device removed or network share dropped; disk full or quota exceeded on write; handle reopened concurrently.

Common situations: Calling Read/Write with a buffer or size not a multiple of the sector size on an unbuffered device; passing an offset/length of 0 or non-sector-multiple; using a stack/unaligned buffer; operating on a device after Close(); removable media ejected mid-run; SMB/network path failure.

Related errors

• Failed to schedule async IO: {}, handle {}
• Failed to schedule work: {}
• Page read from storage failed, skipping page. Inner exceptio
• Slowdown: Unable to add trigger to epoch
• RetrieveCheckpointFile: unexpected state{retStateType}

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/14c2e191a09b9db3. Report an issue: GitHub.