Agenda

  • 100,000 meters
    • What are Coroutines?
  • 10,000 meters
    • QEMU Coroutine API
  • 100 meters
    • Under The Covers: ucontext
  • The problems with Coroutines

The 100,000 Meter View

What are Coroutines?

A way for QEMU to do
cooperative,
concurrent multitasking

Coroutines vs. Threads

  • Threads
    • OS Managed
      • Kernel has a scheduler
      • Preemptive
    • Requires more sophisticated locking

Coroutines vs. Threads

  • Coroutines
    • Userspace Managed
      • Nobody schedules coroutines
      • Essentially just a bunch of non-local gotos
      • Cooperative
    • Lower transactional overhead
What does QEMU use?

QEMU is a hybrid architecture.

Threads are used (worker threads, vcpu threads, etc.),
      in addition to coroutines (mainly in block layer)

Why does QEMU use coroutines?

QEMU uses an event loop; In the past, for long running tasks, we had to rely on callback functions
  • Callbacks cumbersome, and difficult to follow
  • Coroutines offer linear function flow

Where does QEMU use coroutines?

Block layer, of course!


    All block I/O functions are coroutines...

    ... as well as some QMP commands (e.g. Block Jobs)

The 10,000 Meter View

The Coroutine API

States of a Coroutine

The Coroutine Structure







Using Coroutines, Part 1

Core API Functions:

CoQueue

  • Simple a queue of coroutines to be run
  • Queue entries can be scheduled to "wake up"
    • Awoken entries run after yield/terminate

Using Coroutines, Part 2

CoQueue API Functions:

Using Coroutines, Part 3

CoMutex API Functions
(Wait, I thought we were cooperative?!)
  • Still need to protect some sections
    • E.g., image format driver writing into an image
  • qemu_co_mutex_lock()
    • Try to acquire lock
    • If fails: go on end of queue, and yield
  • qemu_co_mutex_unlock()
    • release lock
    • wakeup next co on queue

Using Coroutines, Part 4

Lock API Functions
  • qemu_co_rwlock_rdlock()
    • Waits for no writers
    • While waiting, go on end of queue
  • qemu_co_rwlock_wrlock()
    • Waits for no writers or readers
    • While waiting, go on end of queue
  • qemu_co_rwlock_unlock()
    • wakeup all readers, or one writer

The 100 Meter View

Underneath it all

Coroutines in QEMU

Multiple implementations:
  • ucontext
    • Default, most "mileage"
    • Not on all platforms
  • sigaltstack
    • Newer, as of 2013 (Alex Barcelo)
    • Available on most (all?) POSIX platforms
    • Paper: "Portable Multithreading: The Signal Stack Trick For User-Space Thread Creation", Engelschall

Coroutines in QEMU

Less common implementations:
  • win32
    • Uses Windows Fibers
    • Windows-only
  • gthreads
    • Slow
    • Not really functional
    • Debug only (anyone using it?)

Coroutines Creation - ucontext

Or, fun with trampolines

  • getcontext / makecontext initializes stack
    • coroutine_trampoline
    • current environment saved as uc_link
  • jump on the trampline
    • swapcontext() does the jump
    • jmp_buf saved before coroutine fn call
    • jump back to the caller
  • next siglongjmp to saved jmp_buf will start fn

Coroutines Creation - ucontext 

Coroutine *qemu_coroutine_new(void)
{

/*  [...] */

    arg.p = co;

    makecontext(&uc, (void (*)(void))coroutine_trampoline,
                2, arg.i[0], arg.i[1]);

    /* swapcontext() in, siglongjmp() back out */
    if (!sigsetjmp(old_env, 0)) {
        swapcontext(&old_uc, &uc);
    }
    return &co->base;
}

Problems with Coroutines?

  • Makes debugging painful at times
    • Did I mention trampolines?
    • Check out scripts/qemu-gdb.py
    • "qemu coroutine coroutine ptr"
  • Portability / Platform availability
    • Many methods have limited availability
    • Different implementations for different platforms
  • Coroutine implementation is complex
    • Does make usage easier and less error-prone
  • Can rely on compiler / glibc specific behavior
    • Maybe this is fine 
/*
 * we want to use bottom half because we want to make sure the below
 * sequence of events.
 *
 *   1. Yield the coroutine in the QEMU thread.
 *   2. Submit the coroutine to a worker thread.
 *   3. Enter the coroutine in the worker thread.
 * we cannot swap step 1 and 2, because that would imply worker thread
 * can enter coroutine while step1 is still running
 */
#define v9fs_co_run_in_worker(code_block)                               \
    do {                                                                \
        QEMUBH *co_bh;                                                  \
        co_bh = qemu_bh_new(co_run_in_worker_bh,                        \
                            qemu_coroutine_self());                     \
        qemu_bh_schedule(co_bh);                                        \
        /*                                                              \
         * yield in qemu thread and re-enter back                       \
         * in worker thread                                             \
         */                                                             \
        qemu_coroutine_yield();                                         \
        qemu_bh_delete(co_bh);                                          \
        code_block;                                                     \
        /* re-enter back to qemu thread */                              \
        qemu_coroutine_yield();                                         \
    } while (0)

Questions?