HTTP Client

An application that fetches a URL someone else chose, a webhook target, an avatar, an import link, is one request away from fetching http://169.254.169.254/ on its own behalf. muzak.NewClient builds the client to make those calls with: a net/http client with the decisions that make it safe already made.

client := muzak.NewClient(muzak.ClientOptions{})
defer client.Close()

The zero options are the strict ones. Loopback, private, link-local and metadata addresses are refused, at most five redirects are followed and never from https down to http, at most ten mebibytes of a response are read, a call gives up after thirty seconds, and a request is retried only when sending it twice is harmless.

Making a request

Client.Do takes an ordinary *http.Request and behaves as http.Client.Do does, except where safety calls for something else. The helpers decode JSON for the common case.

// Upstream is the client every handler calls other services with. One client
// for the whole application, because it holds the pool of open connections.
var Upstream = muzak.NewClient(muzak.ClientOptions{})

// UpstreamLifecycle releases the client's idle connections on shutdown.
var UpstreamLifecycle = muzak.NewLifecycle("upstream", nil,
    func(context.Context) error { return Upstream.Close() })

type PreviewIn struct {
    URL string `query:"url"`
}

func (in *PreviewIn) Validate(v *muzak.Validation) {
    // A syntax check, not a safety one: whether the address may be fetched
    // is decided by the client, at the moment it connects.
    v.String(&in.URL).Required().HTTPS()
}

type PageMeta struct {
    Title string `json:"title"`
}

func Preview(ctx *muzak.Context, in PreviewIn) (PageMeta, error) {
    meta, err := core.Upstream.GetJSON[PageMeta](ctx.Context(), in.URL)
    var refused *muzak.AddressRefusedError
    if errors.As(err, &refused) {
        return PageMeta{}, muzak.BadRequest("that address cannot be fetched")
    }
    return meta, err
}

app := muzak.New(muzak.AppOptions{Title: settings.AppName},
    muzak.WithLifecycle(core.UpstreamLifecycle),
)

Passing ctx.Context() matters twice over: the call ends when the request it serves ends, and the request id the RequestID middleware assigned goes along in X-Request-Id, with the trace context when the application traces requests (see Propagation).

Any other error from Do is safe to return from a handler as it is, and renders as an opaque 500. Its message names the method and the origin, GET https://api.example.com, and never the path or the query string, which is where URLs carry their secrets.

What is refused

Every address the client is about to connect to is checked, on the socket, at the moment of connecting. It is not checked when the URL is read, because that is not the address that matters: a name can resolve somewhere else by the time it is fetched.

An IPv6 address that delivers to an IPv4 one is judged by both, so [::ffff:127.0.0.1], [64:ff9b::7f00:1] (NAT64), [2002:7f00:1::] (6to4), [::ffff:0:7f00:1] (the IPv4-translated form of SIIT) and [::127.0.0.1] are refused with the address they reach. The local-use NAT64 block 64:ff9b:1::/48 belongs to a network's own translator, which places the IPv4 address in one of several layouts the client cannot see. It is refused as a private range is, and an address in it is read in every layout a translator may use, so a metadata service or a network in DeniedNetworks stays refused through it even with AllowPrivateNetworks set.

A host that browsers and the C resolver read as an IPv4 address and Go reads as a name is refused whatever it points at: 127.1, 0177.0.0.1, 0x7f.0.0.1, 2130706433. A check and a fetch that disagree about what a host is are how request forgery filters are usually got around, so the client does not guess. localhost and every name under it are refused with loopback.

A refusal is an *muzak.AddressRefusedError naming the host, the address and what it is. Nothing was sent, and it is never retried.

Names, rebinding and redirects

The check runs in the dialer, which is the one place every connection is opened, so:

• a name that resolves to a private address is refused when it is dialled;
• a name that resolves to a public address and then, the second time it is asked, to the metadata service, which is what DNS rebinding is, is refused on the second connection;
• a name that resolves to several addresses is connected only to the ones that pass, in the resolver's order;
• a redirect, however many hops in, meets the same check, and so does every retry.

Calling your own services

Service-to-service calls inside one network are what AllowPrivateNetworks is for. It never opens a metadata service, which holds the credentials of the machine the client runs on.

// Calls to the inventory service, inside the cluster. Only that subnet.
inventory := muzak.NewClient(muzak.ClientOptions{
    AllowedNetworks: []netip.Prefix{netip.MustParsePrefix("10.20.0.0/16")},
})

// Calls to any internal service, but never to the database subnet.
internal := muzak.NewClient(muzak.ClientOptions{
    AllowPrivateNetworks: true,
    DeniedNetworks:       []netip.Prefix{netip.MustParsePrefix("10.30.0.0/16")},
})

DeniedNetworks wins over everything, AllowedNetworks over everything else. Keep a strict client for URLs that came from outside and a relaxed one for your own configuration, rather than one relaxed client for both. A prefix that is not valid panics in NewClient, because a list with a hole in it does not mean what it says.

Behind an egress proxy

The client never reads HTTP_PROXY. A proxy decides where requests go, and an environment variable is a setting anything that can set one controls. Name it instead:

egress, _ := url.Parse("http://egress.internal:3128")
client := muzak.NewClient(muzak.ClientOptions{Proxy: egress})

The proxy is connected to without the address checks, and is trusted to apply its own: it is the proxy that resolves the names it is asked for. The client still refuses what it can judge without resolving anything, a refused address written literally, a localhost name, a name ending in a number, and a name outside ASCII, which net/http would map before the proxy saw it. A proxy that enforces no egress policy of its own reopens every request forgery the client otherwise closes.

Bounds

A negative timeout removes that bound and leaves the request's context in charge. The body limit cannot be removed, only raised. A response that declares a Content-Length over the limit is refused by Do with muzak.ErrResponseTooLarge before any of it is read; one that declares none, or is compressed, fails with the same error on the read that passes it, so a gzip body that expands a thousandfold stops at the limit.

Timeout keeps running after Do returns, until the body is closed or read to its end. Close every body, as with net/http.

Redirects

A redirect is followed only to an address the client would have connected to directly, never from https to plain http, and at most MaxRedirects times. MaxRedirects: -1 follows none and hands back the redirect itself, Location and all.

When a redirect leaves the origin the request was sent to, Authorization, Cookie, Proxy-Authorization and every header in SensitiveHeaders are removed, and so is the Referer net/http would otherwise add. An origin is the scheme, host and port together, so a subdomain, another port and an upgrade from http to https all count as leaving it, which is stricter than net/http. Once one hop has left, the headers stay off for the rest of the chain, so a host passed through cannot send the request back to the origin, with the credentials, at a path of its choosing.

client := muzak.NewClient(muzak.ClientOptions{
    SensitiveHeaders: []string{"X-Api-Key"},
})

The headers are what is removed, not the body. A 307 or 308 asks for the same request again, body included, and the client follows it to another origin as net/http does, so a request body goes wherever such a redirect points, as long as the address passes. For a call whose body must reach one origin and no other, set MaxRedirects: -1 and judge the redirect yourself.

Retries

A request is retried only when sending it twice is harmless: GET, HEAD, OPTIONS, PUT and DELETE, and any request carrying an Idempotency-Key header (or X-Idempotency-Key, which net/http's own transport also honours). It is retried after a connection that failed or broke, and after 429, 502, 503 and 504. When the attempts run out, the last response is returned as it is, for the caller to read.

What is never retried: a POST or PATCH without an idempotency key, a request whose body cannot be read twice (it has no GetBody, which http.NewRequest sets for a bytes.Reader, bytes.Buffer or strings.Reader), a refused address, a refused redirect, an open circuit, a certificate that does not verify, a TLS alert, a name that does not exist, and a call whose context has ended.

req, _ := http.NewRequestWithContext(ctx.Context(), http.MethodPost, chargesURL, bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
// The payment provider deduplicates on this key, so a retry cannot charge twice.
req.Header.Set("Idempotency-Key", order.ID)
charge, err := client.DoJSON[Charge](req)

The wait between attempts is random, between zero and a ceiling that starts at RetryBaseDelay (100 ms) and doubles up to RetryMaxDelay (2 s). Drawing the whole wait at random is what keeps a thousand clients that failed together from retrying together.

Retry-After

A Retry-After header, in seconds or as an HTTP date, is a floor on the wait. One that asks for longer than MaxRetryAfter (10 s) is not waited out: the response is returned, since retrying sooner than the server asked is what it asked not to have. A wait that would outlast the call's deadline is not slept either; the response is returned at once rather than after the deadline has been spent sleeping. A date in the past means now, and a number too large to represent is treated as a very long wait rather than wrapping into a negative one.

The retry budget

Retries are what turn an upstream's bad minute into its bad hour: every failing request becomes three, and the load that made it fail triples. The client keeps a bucket of retry tokens. Every response that is not a server error earns a tenth of one, every retry spends one, and a retry with no token to spend is not made. It starts with ten and holds at most ten.

client := muzak.NewClient(muzak.ClientOptions{
    RetryBudget: muzak.RetryBudget{Ratio: 0.2, Burst: 20},
})

MaxAttempts: 1 turns retrying off altogether.

Circuit breaker

A host that is down answers slowly or not at all, and every caller waiting on it holds a goroutine and a connection for as long as the timeout allows. The breaker is off until it is given a threshold:

client := muzak.NewClient(muzak.ClientOptions{
    CircuitBreaker: muzak.CircuitBreakerOptions{Threshold: 5, Cooldown: 30 * time.Second},
})

After Threshold failures in a row to one host, a connection error, a timeout or a 5xx, requests to that host fail at once with muzak.ErrCircuitOpen for Cooldown. Then one request is let through: success closes the circuit, failure opens it for another cooldown, and every other request is refused while that one is out. A caller cancelling its own request is not a failure; a deadline passing is, since a host that hangs is the one a breaker is most for.

A host is a scheme, a name and a port, and each step of a redirect counts against the host it reached. Only failing hosts are remembered, at most MaxHosts (1024) of them, the least recently used forgotten first, so calling ten thousand hosts cannot grow the table.

Propagation

Every request is passed through ClientOptions.Propagate once before it is sent. The default, muzak.DefaultPropagate, copies the request id from the context into X-Request-Id, so that the logs of two services can be joined on it, and leaves a header the caller set alone. An id that holds a control character or is longer than 256 bytes is left out rather than failing the call. When the application traces requests, it also writes the W3C traceparent and tracestate of the current span with muzak.InjectTraceContext, so the service called records its work as part of the same trace; a traceparent the caller set is left alone, and a request that is not traced sends none. See Observability.

client := muzak.NewClient(muzak.ClientOptions{
    Propagate: func(ctx context.Context, h http.Header) {
        muzak.DefaultPropagate(ctx, h)
        h.Set("X-Tenant", tenantFrom(ctx))
    },
})

A function that does nothing turns propagation off. Whatever a propagator adds is checked like any other header. On a call made with a typed endpoint's Call, a propagator may fill a header the input binds and left empty, but one that changes a header the input set refuses the call; see Typed Endpoints.

DefaultPropagate adds its headers to every request, whoever the URL names. Neither header is a secret, but both let the receiver join its records to yours, and a traceparent says whether you are sampling. A client that fetches URLs from outside, a webhook target or an avatar, should not hand them to whoever chose the URL: give that client a Propagate that adds nothing, and keep the default for the clients that call your own services.

// Fetches URLs users supply. Nothing about our own requests goes with them.
fetcher := muzak.NewClient(muzak.ClientOptions{
    Propagate: func(context.Context, http.Header) {},
})

Headers

A header value holding a carriage return, a line feed or a NUL is refused before anything is sent, and so is a header name that is not a token. That is how a value that came from outside, a forwarded token or an identifier, would otherwise write a header of its own. The error names the header and never repeats its value.

JSON

A status that is not 2xx is an *muzak.RemoteError carrying the status, the header and the first 64 KiB of the body. When the body is the error envelope a Muzak application writes, the default one or a problem details document, its Code, Message, Details and RequestID are filled in, and Code is what to branch on. The body is read as an envelope only under a JSON media type, whole or not at all, keeping at most a hundred details, so a truncated or malformed body leaves those fields empty rather than half filled. Its message is the method, origin and status, muzak: GET https://api.example.com answered with status 404 (not_found), naming the code only when it is a plain identifier of at most 64 bytes, and never the body or the message, which are the other server's text. Handle it with errors.As and the StatusCode or Code, not by matching the message.

item, err := client.GetJSON[Item](ctx.Context(), itemURL)
var remote *muzak.RemoteError
if errors.As(err, &remote) && remote.Code == "not_found" {
    return Item{}, muzak.NotFound("no such item upstream")
}

A success not labelled as JSON is refused rather than guessed at, which is how an HTML error page from a proxy announces itself. 204 and HEAD decode to the zero T. Members T does not have are ignored, as a client reading another service's answers should; duplicate members and invalid UTF-8 are refused.

Errors

Shutting down

Close releases the idle connections and does not stop the client: a request made afterwards opens a connection again, which is what lets a Lifecycle's Stop call it while a late request is still finishing.

Testing code that calls out

A test server from httptest listens on 127.0.0.1, which a strict client refuses. Build the client the code under test uses from options the test can set:

func TestPreview(t *testing.T) {
    upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        _, _ = w.Write([]byte(`{"title":"Plumbus"}`))
    }))
    defer upstream.Close()

    client := muzak.NewClient(muzak.ClientOptions{AllowPrivateNetworks: true})
    defer client.Close()
    meta, err := client.GetJSON[PageMeta](t.Context(), upstream.URL)
    // ...
}

Keep AllowPrivateNetworks out of the production configuration of a client that fetches URLs from outside, and test the refusal too: a URL of http://127.0.0.1/ must come back as an *muzak.AddressRefusedError. Testing covers testing the application's own routes.

The same applies to a JWT bearer scheme's key set, which is fetched through a client built here with the strict defaults. An identity provider on a private network, or a test server, needs JWKSOptions.Client set to a client with AllowPrivateNetworks or an AllowedNetworks entry; see Authentication.

Limits

• A NAT64 translator configured with a prefix of the network's own choosing, outside 64:ff9b::/96 and 64:ff9b:1::/48, is invisible to the client: its addresses look like any public IPv6 address. If your network has one, list that prefix in DeniedNetworks.
• A 307 or 308 that leaves the origin is followed with the request body, as described under Redirects.
• DefaultPropagate sends the request id and the trace context to any URL, as described under Propagation.

Options reference

NewClient panics on options that cannot mean what they say: an invalid prefix in AllowedNetworks or DeniedNetworks, a Proxy with no host or a scheme no proxy speaks, and a BaseURL that is not a base.

Where to go next

Typed Endpoints covers calling another Muzak service with the server's own types through this client, Safe Defaults lists every other default and how to relax it, Lifecycle covers closing the client on shutdown, Logging covers the request id the client carries along, and Observability covers the trace it continues.