Behind a Proxy

Muzak attributes a request to the peer that opened the connection, and ignores every forwarding header. That is the only safe default: a forwarding header is a request header like any other, and a server that believes one without knowing who wrote it lets any client claim any address it likes.

Behind a proxy the default is wrong in the other direction, since every request then appears to come from the proxy. Naming the proxy is what makes the header believable.

Naming the proxy

app := muzak.New(muzak.AppOptions{
    Title: settings.AppName,
    Addr:  settings.Addr,
    // Which address a request is attributed to. Nothing is believed from a
    // header until the proxy that wrote it is named here.
    ClientIP: muzak.ClientIPOptions{TrustedProxies: settings.TrustedProxies},
})

type Settings struct {
    // TrustedProxies lists the proxies whose X-Forwarded-For header is
    // believed, as a comma-separated list of addresses or CIDR prefixes. It is
    // empty by default, so every request is attributed to the peer that made
    // it: a forwarding header is written by whatever sent the request, and
    // believing one from an unknown sender hands every client the ability to
    // choose which budget it spends.
    TrustedProxies []string `env:"TRUSTED_PROXIES"`
}

# Proxies whose X-Forwarded-For header is believed, as a comma-separated list of
# addresses or CIDR prefixes. Leave it empty when nothing sits in front of this
# service.
TRUSTED_PROXIES=10.0.0.0/8,172.16.0.0/12

Entries are plain addresses (10.1.2.3) or CIDR prefixes (10.0.0.0/8), and both address families are accepted. An entry that cannot be parsed is reported when the application is built, rather than quietly widening or narrowing the policy.

The header is consulted only for a request whose peer is trusted. A request arriving directly from the internet with an X-Forwarded-For of its own choosing is attributed to the address it actually came from.

A header other than X-Forwarded-For

muzak.ClientIPOptions{
    TrustedProxies: []string{"10.0.0.0/8"},
    Header:         "CF-Connecting-IP",
}

DefaultForwardedHeader is X-Forwarded-For. Set Header to whatever your proxy actually writes, such as CF-Connecting-IP or X-Real-IP.

Reading the address

ctx.ClientIP()   // the address as a string, normalised
ctx.ClientAddr() // the same as a net/netip.Addr, for comparing against a prefix

addr := ctx.ClientAddr()
if !addr.IsValid() {
    // The connection has no address that can be parsed, which happens on a
    // listener that is not addressed by IP, such as a Unix socket.
}
if office.Contains(addr) {
    // ...
}

The result is normalised, so an address written in IPv4-in-IPv6 form and the same address written plainly are one value rather than two. That is what stops a client from being counted twice, or from evading a count, by rewriting its own address.

ClientIP returns the empty string when the connection has no parsable address. Code that keys on the result has to handle that, which is what muzak.IPTracker does by refusing the request rather than counting every such request under one shared key.

What depends on getting this right

Feature	Uses the client address for
Rate limiting	`IPTracker` and `IPPrefixTracker` key the budget on it
WebSockets	`MaxConnectionsPerIP` bounds one client's share of the process
Server-sent events	`MaxStreamsPerIP` does the same for streams
Anything you write	Deny lists, audit logs, geo decisions

With an untrusted proxy in front and nothing named in TrustedProxies, every request carries the proxy's address, so the whole world shares one rate limit budget and one connection allowance. With a wrongly trusted header, any client picks its own budget by writing a header. Both failure modes are why this is configuration rather than a guess.

Rate limiting behind a proxy

Once the address is right, the tracker follows.

muzak.WithRateLimit(muzak.RateLimitOptions{
    Tracker: muzak.IPPrefixTracker(32, 64),
    Quotas: []muzak.Quota{
        {Name: "short", Window: time.Second, Limit: 3},
        {Name: "long", Window: time.Minute, Limit: 100},
    },
})

An IPv6 /64 is the block size most providers hand out, so keying on the exact address gives a client holding one a fresh budget on every request. IPPrefixTracker(32, 64) keeps IPv4 exact and collapses IPv6 to the allocation it came from. See Rate Limiting.

A service running more than once needs a storage the processes share, or a limit of a hundred a minute becomes a hundred a minute per process.

TLS terminated in front

Common, and it changes three things:

The service speaks plain HTTP, so CertFile, KeyFile and TLSConfig stay empty.
Cookies still need Secure: true, because the browser leg of the connection is HTTPS and that is the leg the attribute governs.
Strict-Transport-Security, if you set it, cannot be gated on r.TLS != nil, because the request Muzak sees is not the encrypted one. Gate it on whatever header the proxy writes, or set it at the proxy.

See TLS.

Streaming through a proxy

An event stream is the request most likely to be broken by something in the middle. Muzak sets Cache-Control: no-cache, no-transform and X-Accel-Buffering: no, and writes a keepalive comment every SSEOptions.KeepAlive, so a proxy does not close a connection it believes to be idle.

Check the proxy's own settings too. proxy_buffering off and a read timeout longer than the keepalive are what nginx needs; the equivalents exist elsewhere. A WebSocket needs the proxy configured to upgrade the connection at all.

Request identifiers

Muzak generates an identifier per request and ignores an inbound X-Request-Id, because an attacker-controlled identifier is an attacker-controlled log field.

Where a trusted proxy assigns the identifier and you want the two logs to agree, accept it:

app := muzak.New(muzak.AppOptions{
    Title:                "Awesome API",
    TrustRequestIDHeader: true,
})

An inbound value is honoured only if it parses as a UUID, so it can never carry newlines or control characters into a log line. Turn it on only where nothing untrusted can reach the service directly, since the header would otherwise come straight from the client.

Body limits in two places

The proxy has its own body limit, and so does Muzak. Set the proxy's at or above the route's, or a large upload is refused with the proxy's error page rather than the application's 413 and its error envelope. See File Uploads.

Checking it works

curl -s http://localhost:8080/whoami -H 'X-Forwarded-For: 203.0.113.9'

type WhoAmIOut struct {
    ClientIP string `json:"client_ip"`
}

r.Get("/whoami", func(ctx *muzak.Context, _ muzak.Empty) (WhoAmIOut, error) {
    return WhoAmIOut{ClientIP: ctx.ClientIP()}, nil
}, muzak.Hidden())

Called directly, that answers with your own address whatever the header says. Called through a proxy named in TrustedProxies, it answers with what the proxy reported. Those two results are the whole configuration, and they are worth checking once per deployment.

Remove the route afterwards, or leave it Hidden and behind a guard.

Where to go next

Server Configuration covers the listener itself, and Rate Limiting covers the feature that depends on this most.