Background Tasks

Some work belongs to a request without belonging to its response: the welcome email after a sign-up, an audit record, a cache to warm. The client should not wait for it, and a bare go func() is the wrong tool, because nothing bounds how many there are, nothing waits for them at shutdown, and the one that captured the *muzak.Context reads another request's values a moment later. Context.AfterResponse registers the work instead.

func SignUp(ctx *muzak.Context, in schemas.SignUpIn) (schemas.UserOut, error) {
    user, err := users.Create(ctx.Context(), in)
    if err != nil {
        return schemas.UserOut{}, err
    }

    // Copied out before registering: the task never sees ctx.
    log := ctx.Logger()
    email := user.Email
    if err := ctx.AfterResponse(func(bg context.Context) {
        if err := mailer.SendWelcome(bg, email); err != nil {
            log.ErrorContext(bg, "welcome email failed", "error", err)
        }
    }); err != nil {
        // The queue is full or the server is shutting down. The user exists
        // either way; the email is what is lost, so say so.
        log.Warn("welcome email not queued", "error", err)
    }
    return schemas.UserOut{ID: user.ID, Email: user.Email}, nil
}

app := muzak.New(muzak.AppOptions{
    Title: "Awesome API",
    Background: muzak.BackgroundOptions{
        Workers: 8,   // tasks running at once
        Queue:   256, // tasks waiting for a worker
    },
})

When a task runs

Only when the handler succeeds. A task runs when the handler returned a nil error without panicking. A request that fails drops every task it registered, whether its handler returned an error, a guard or the binder refused it, or something panicked, including a value that panicked while its response was being encoded. A task registered by a guard or a provider follows the same rule: it runs if the route's handler then succeeds.

After the response. Tasks are handed to the workers once the response has been passed to net/http, so the response never waits for them. They are handed over before the middleware installed with App.Use has unwound, and a task can begin before the last byte has reached the client.

Whatever happens to the response afterwards. Once the handler has returned nil, its own work is done: a client that leaves before reading the response does not cancel the email that was promised to it, and neither does a response value whose MarshalJSON returns an error. A value that panics while it is encoded is the exception, because something panicked.

Unless the route's releases were told it failed. On a route with an Acquire provider, a value that fails to encode is reported to each Release as a failure, as an error of the handler's own would be. The work the tasks were to follow has been undone, a Transaction rolled back, so the tasks are dropped with it. The tasks follow what the releases were told.

In no particular order. The tasks of one request run concurrently with each other and with everyone else's. A task that has to follow another should be one task.

What a task receives

The context is cancelled when the shutdown deadline is reached, with a cause that says so, which is worth logging when a task gives up:

if err := export.Run(bg); err != nil {
    log.ErrorContext(bg, "export abandoned", "error", err, "cause", context.Cause(bg))
    // cause="muzak: the shutdown deadline passed before this background task finished"
}

A task that honours it is the difference between a shutdown that finishes on time and one that has to abandon the task.

The *muzak.Context is pooled and belongs to another request by the time the task runs, so the task never receives it; copy out what it needs before registering it. ctx.Logger() returns a logger that carries the request's method, path and identifier (and its trace when tracing is on) and holds nothing of the pooled Context, so it is the one value always worth capturing.

Bounds

A task counts against Queue from the moment it is registered, while its request is still being served, so a request that succeeds never finds the queue full at the moment its tasks are handed over. A registration past the bound returns muzak.ErrBackgroundQueueFull at once: the handler is never blocked waiting for room, and no goroutine is started to hold the task. What to do then is the handler's decision: do the work before returning, drop it with a log line, or fail the request.

The same rule has a cost: a slow request holds the place its task took until the request ends. Many slow requests that each registered a task early can fill the queue while no worker is busy, and later registrations are refused until they finish. Register a task as late in the handler as you can, once the work it follows is done, and size Queue for the requests in flight as well as for the tasks waiting.

There is no unbounded setting, on purpose: a queue without a cap is a backlog anyone who can send requests can grow. A negative value is a build error and zero takes the default.

No goroutine exists while there is nothing to run. A worker is started when a task is handed over and fewer than Workers are running, and exits as soon as the queue is empty, so an application that never registers a task never runs one, and a request that registers none pays nothing for the feature.

A panic in a task is recovered and logged with the request identifier and the stack, and the worker carries on with the next task.

14:32:07.913 ERROR [Server]        muzak: recovered from a panic in a background task  panic="assignment to entry in nil map"  request_id=0192a7c1  stack=...

Shutdown

A shutdown runs the tasks it already accepted, in an order that keeps them working:

1. Readiness reports unavailable, and the server keeps serving through DrainDelay (see Health Checks). Tasks registered meanwhile are accepted as usual.
2. The listeners close. From here on AfterResponse returns muzak.ErrBackgroundShuttingDown, while requests in flight are drained; the tasks they registered earlier are still handed over when they finish.
3. The queued and running tasks are waited for.
4. The lifecycle components are stopped, so a task can still use the database pool it needs.

All of it shares ShutdownTimeout. When the deadline passes during step 3, the tasks still waiting for a worker are dropped, the running ones have their context cancelled and are given 100 milliseconds to return, and the components are stopped anyway. Both are logged:

14:32:56.903 WARN  [Server]        Shutdown deadline reached; dropped 12 queued background tasks that had not started
14:32:57.004 WARN  [Server]        1 background task still running after the shutdown deadline and its grace period; stopping lifecycle components anyway

Size the pool with that in mind: Workers should get through a full Queue within ShutdownTimeout. A negative ShutdownTimeout waits for the tasks without limit.

Where it is refused

An application served by hand through App.ServeHTTP runs its tasks the same way, but nothing drains them, since no Shutdown is involved.

Testing

The test client serves the application through the same server and shuts it down the same way when the test ends, so the test's cleanup waits for the tasks its requests registered. A test that wants to see a task's effect before then waits for it on a channel the task closes.

Where to go next

Lifecycle covers the components a task uses, Health Checks the readiness drop and drain delay that begin a shutdown, and Timeouts the deadline a task does not inherit.