Behind a Proxy

Muzak attributes a request to the peer that opened the connection, and ignores every forwarding header. That is the only safe default: a forwarding header is a request header like any other, and a server that believes one without knowing who wrote it lets any client claim any address it likes.

Behind a proxy the default is wrong in the other direction, since every request then appears to come from the proxy. Naming the proxy is what makes the header believable.

Naming the proxy

app := muzak.New(muzak.AppOptions{
    Title: settings.AppName,
    Addr:  settings.Addr,
    // Which address a request is attributed to. Nothing is believed from a
    // header until the proxy that wrote it is named here.
    ClientIP: muzak.ClientIPOptions{TrustedProxies: settings.TrustedProxies},
})

type Settings struct {
    // TrustedProxies lists the proxies whose X-Forwarded-For header is
    // believed, as a comma-separated list of addresses or CIDR prefixes. It is
    // empty by default, so every request is attributed to the peer that made
    // it: a forwarding header is written by whatever sent the request, and
    // believing one from an unknown sender hands every client the ability to
    // choose which budget it spends.
    TrustedProxies []string `env:"TRUSTED_PROXIES"`
}

# Proxies whose X-Forwarded-For header is believed, as a comma-separated list of
# addresses or CIDR prefixes. Leave it empty when nothing sits in front of this
# service.
TRUSTED_PROXIES=10.0.0.0/8,172.16.0.0/12

Entries are plain addresses (10.1.2.3) or CIDR prefixes (10.0.0.0/8), and both address families are accepted. An entry that cannot be parsed is reported when the application is built, rather than quietly widening or narrowing the policy. A prefix written in the IPv4-in-IPv6 form, such as ::ffff:10.0.0.0/104, means the IPv4 prefix it wraps (10.0.0.0/8). It used to parse and then match nothing, leaving the proxy it named untrusted without a message.

The header is consulted only for a request whose peer is trusted. A request arriving directly from the internet with an X-Forwarded-For of its own choosing is attributed to the address it actually came from.

How the header is read

The walk goes right to left through the forwarding header, which is the only direction that cannot be lied to. Each hop appends the address it saw, so the rightmost entries were written by the proxies nearest this server, and the leftmost by whoever sent the request, who may have invented them. Each trusted hop is stepped over, and the first entry that is not a trusted proxy is the client. If every hop was trusted, the leftmost entry is as close to the client as this server can see.

• A request arriving from an untrusted peer is attributed to that peer, whatever the header says.
• An entry that cannot be read (an obfuscated identifier, a mangled address) stops the walk: it hides whatever is to its left, so the nearest hop that was understood is the most that can be claimed. A port after an address, and the brackets IPv6 needs when there is one, are tolerated.
• At most 64 entries are read, so a header of ten thousand addresses is not parsed in full on the way to deciding who sent it.
• The address is normalised: an IPv4-in-IPv6 address is the IPv4 it carries, and an IPv6 zone (fe80::1%eth0) is dropped. A zoned address was never inside a trusted prefix, so a trusted proxy that reported itself with a zone was taken for the client, and one client could mint any number of keys by appending %a, %b and so on to its own address.

A header other than X-Forwarded-For

muzak.ClientIPOptions{
    TrustedProxies: []string{"10.0.0.0/8"},
    Header:         "CF-Connecting-IP",
}

DefaultForwardedHeader is X-Forwarded-For. Set Header to whatever your proxy actually writes, such as CF-Connecting-IP or X-Real-IP.

The standard Forwarded header

muzak.ClientIPOptions{
    TrustedProxies: []string{"10.0.0.0/8"},
    Header:         "Forwarded",
}

Forwarded is read as RFC 7239 defines it: the address is the for parameter of each entry, as in for=203.0.113.9;proto=https or for="[2001:db8::1]:443", and the other parameters are skipped. Parameter names in any case, quoted strings, several entries and several header lines are all read, and the walk is the same right-to-left one, under the same trust rule, as for X-Forwarded-For. An entry whose for is unknown or an obfuscated identifier such as _hidden, or that has no for at all, stops the walk at the nearest trusted hop. A comma or semicolon inside a quoted parameter is data, so a proxy that quotes a client's Host into its own entry cannot be talked into a client's for.

Before 0.2.9 the header was accepted but its entries could not be read, so every client behind the proxy resolved to the proxy and shared one budget.

Reading the address

ctx.ClientIP()   // the address as a string, normalised
ctx.ClientAddr() // the same as a net/netip.Addr, for comparing against a prefix

addr := ctx.ClientAddr()
if !addr.IsValid() {
    // The connection has no address that can be parsed, which happens on a
    // listener that is not addressed by IP, such as a Unix socket.
}
if office.Contains(addr) {
    // ...
}

The result is normalised, so an address written in IPv4-in-IPv6 form and the same address written plainly are one value rather than two. That is what stops a client from being counted twice, or from evading a count, by rewriting its own address.

ClientIP returns the empty string when the connection has no parsable address. Code that keys on the result has to handle that, which is what muzak.IPTracker does by refusing the request rather than counting every such request under one shared key.

What depends on getting this right

With an untrusted proxy in front and nothing named in TrustedProxies, every request carries the proxy's address, so the whole world shares one rate limit budget and one connection allowance. With a wrongly trusted header, any client picks its own budget by writing a header. Both failure modes are why this is configuration rather than a guess.

Rate limiting behind a proxy

Once the address is right, the tracker follows. The default IPTracker counts an IPv4 address exactly and an IPv6 address by its /56, the block a provider delegates to one home or small site, so a client that rotates addresses inside its own range still spends one budget.

muzak.WithRateLimit(muzak.RateLimitOptions{
    Quotas: []muzak.Quota{
        {Name: "short", Window: time.Second, Limit: 3},
        {Name: "long", Window: time.Minute, Limit: 100},
    },
})

ClientIPOptions.ConnectionIPv6Prefix changes the prefix for the tracker and for the per-client connection caps together. Set 64 where many unrelated users share one /56, which also loosens the connection caps, or 48 where sites are delegated a /48. For one route, IPPrefixTracker(32, 64) keeps IPv4 exact and collapses IPv6 to the length you give it. A proxy that already bounds per-address traffic can set Tracker: IPPrefixTracker(32, 128) to count every IPv6 address on its own. See Rate Limiting.

A service running more than once needs a storage the processes share, or a limit of a hundred a minute becomes a hundred a minute per process.

TLS terminated in front

Common, and it changes four things:

• The service speaks plain HTTP, so CertFile, KeyFile and TLSConfig stay empty.
• Cookies still need Secure: true, because the browser leg of the connection is HTTPS and that is the leg the attribute governs.
• SecurityHeaders sends Strict-Transport-Security only when r.TLS != nil, so behind a proxy that terminates TLS it is not sent: the request Muzak sees is not the encrypted one. Set it at the proxy, or in middleware gated on whatever header the proxy writes.
• AppOptions.RedirectHTTPS cannot see the client's scheme either, so it believes the X-Forwarded-Proto of a proxy that TrustedProxies names (or the proto of Forwarded, when that is ClientIPOptions.Header) and ignores the claim from any other peer. AppOptions.AllowedHosts never reads X-Forwarded-Host, so list the Host the proxy passes on.

See TLS and Allowed Hosts and HTTPS.

Streaming through a proxy

An event stream is the request most likely to be broken by something in the middle. Muzak sets Cache-Control: no-cache, no-transform and X-Accel-Buffering: no, and writes a keepalive comment every SSEOptions.KeepAlive, so a proxy does not close a connection it believes to be idle.

Check the proxy's own settings too. proxy_buffering off and a read timeout longer than the keepalive are what nginx needs; the equivalents exist elsewhere. A WebSocket needs the proxy configured to upgrade the connection at all.

A keepalive arrives every interval since 0.2.8. Before, it could arrive at 15, 45 and 60 seconds when asked for every 15, twice the silence an idle timeout was meant to be kept off with. And a stream that a proxy would otherwise hold open for as long as its client keeps reading can be given a MaxLifetime, so a client that reads nothing is not held until the server stops. See Server-Sent Events.

For one platform's measured behaviour, including the address its proxy presents and what changes between HTTP/1 and h2c, see Running on Cloud Run.

Request identifiers

Muzak generates an identifier per request and ignores an inbound X-Request-Id, because an attacker-controlled identifier is an attacker-controlled log field.

Where a trusted proxy assigns the identifier and you want the two logs to agree, accept it:

app := muzak.New(muzak.AppOptions{
    Title:                "Awesome API",
    TrustRequestIDHeader: true,
})

An inbound value is honoured only if it parses as a UUID, so it can never carry newlines or control characters into a log line. Turn it on only where nothing untrusted can reach the service directly, since the header would otherwise come straight from the client.

Trace context

With tracing on, a request's traceparent is the same question asked of a trace: by default any valid one is continued, sampling decision included, so a client can attach its requests to another trace or have every one of them sampled. For a service the internet reaches, set TracingOptions.Parent to muzak.TraceParentFromTrustedProxies, which continues a trace only for a connection from an address in ClientIPOptions.TrustedProxies:

app := muzak.New(muzak.AppOptions{
    Title:    "Awesome API",
    ClientIP: muzak.ClientIPOptions{TrustedProxies: settings.TrustedProxies},
    Tracing: muzak.TracingOptions{
        Tracer: exporter,
        Parent: muzak.TraceParentFromTrustedProxies,
    },
})

That is only as good as the proxy. One that passes a client's traceparent on unchanged, as many load balancers do, passes the client's choice on with it, so the proxy has to replace or remove the header; where it cannot, use muzak.TraceParentIgnore, which always starts a new trace. See Observability.

Body limits in two places

The proxy has its own body limit, and so does Muzak. Set the proxy's at or above the route's, or a large upload is refused with the proxy's error page rather than the application's 413 and its error envelope. See File Uploads.

Muzak refuses a body whose declared Content-Length is over the route's limit before it reads any of it. A proxy that holds the whole request until it is complete, as Cloud Run does over HTTP/1, still takes the upload first: the early refusal cannot spare bytes a proxy has already buffered.

Checking it works

curl -s http://localhost:8080/whoami -H 'X-Forwarded-For: 203.0.113.9'

type WhoAmIOut struct {
    ClientIP string `json:"client_ip"`
}

r.Get("/whoami", func(ctx *muzak.Context, _ muzak.Empty) (WhoAmIOut, error) {
    return WhoAmIOut{ClientIP: ctx.ClientIP()}, nil
}, muzak.Hidden())

Called directly, that answers with your own address whatever the header says. Called through a proxy named in TrustedProxies, it answers with what the proxy reported. Those two results are the whole configuration, and they are worth checking once per deployment.

Remove the route afterwards, or leave it Hidden and behind a guard.

Where to go next

Server Configuration covers the listener itself, and Rate Limiting covers the feature that depends on this most.