API Compatibility

A committed OpenAPI document is a promise to every client written against it. Muzak compares the document your application generates now with a baseline you keep in the repository, and fails a test when a change would break one of those clients: an operation that is gone, a request member that became required, a response member that may now be null.

package main

import (
    "testing"

    "muzak.dev/framework/testclient"
)

func TestAPICompatibility(t *testing.T) {
    testclient.AssertCompatible(t, buildApp(), "testdata/openapi.json")
}

The comparison knows how Muzak describes an API, so it judges each change the way a client meets it rather than as a difference of text. Reordering members, renaming a component or adding an optional request member passes; dropping a response member does not.

Recording the baseline

The first run fails, because there is nothing to compare with yet, and says how to create it:

testclient: there is no OpenAPI baseline at testdata/openapi.json; run the test once with
MUZAK_UPDATE_OPENAPI=1 to record the application's current document there, and commit the file

MUZAK_UPDATE_OPENAPI=1 go test -run TestAPICompatibility ./...
git add testdata/openapi.json

With MUZAK_UPDATE_OPENAPI=1 set (exactly 1), the test records the current document instead of comparing, and logs the breaking and possibly breaking changes the new baseline accepts, so the run that accepts a change also says what it accepted. The file is what Document.Marshal writes, indented with sorted keys, so its diff in a pull request is readable.

The baseline is written to the path given and nowhere else. The directories it needs are created, the document is written to a temporary file beside it and renamed into place, so an interrupted run never leaves half a document behind, and a symbolic link at the path is replaced rather than followed. The file is readable by everyone (0644), as a file in a repository is.

What a run reports

A failure lists each change on one line, with where it is and what kind it is:

testclient: the API breaks clients written against the baseline at testdata/openapi.json:
Breaking (2):
  /components/schemas/ItemInput/properties/sku: In requests, the member "sku" is new and required, which a client written against the old document does not send. [request-property-added]
  /paths/~1items~1{id}/delete: The operation DELETE /items/{id} is no longer served. [operation-removed]
If the change is intended, run the test with MUZAK_UPDATE_OPENAPI=1 to record the new document as the baseline.

When a change is intended, record the new baseline and commit it with the change. The baseline's diff is then the part of the pull request a reviewer reads to see what clients will notice.

How a change is judged

Every schema is judged by the direction it travels in.

• What a client sends may only widen. The server has to keep accepting every request it accepted before. A new optional member, a relaxed bound or a longer enum is compatible; a new required member, a narrower type, a tighter bound, a new pattern or format, or null no longer being accepted is breaking.
• What a client reads may only narrow. The client has to keep understanding every response. A member that is gone, that may now be absent, or that may now be null, and a changed type, are breaking; a new member is compatible, because a response is left open.

So the same change can be judged both ways. A member that became optional is compatible in a request, where a client may now leave it out, and breaking in a response, where a client relied on it being there.

Schemas

Each kind starts with request- or response-, the direction it was judged in.

A few of these deserve a sentence:

• A removed request member breaks clients because every object Muzak reads a request into says additionalProperties: false: the decoder refuses a member it does not know, so a client that still sends it gets a 422. On a route registered with AllowUnknownFields the member is ignored instead, which is possibly breaking.
• A new value in a response enum is possibly breaking. A client that switches over the values exhaustively, as generated TypeScript often does, breaks; one that stores the string does not.
• A relaxed response bound is possibly breaking. Only a client that validates what it reads refuses the values it now admits.
• int32 to int64 is a widening. A client generator picks the width of a number by its format, so it is compatible in a request and breaking in a response.
• A changed pattern is treated as a narrowing in a request, since whether one regular expression admits everything another does cannot be decided in general.
• A pattern moved into allOf beside a second one, which Muzak does when two rules ask for a pattern each, is compared as the one new pattern it is, not as a keyword that went away.

Operations

A parameter's value travels as text, so ?id= going from an integer to a string widens it. A form value is compared the same way.

A response header is read the way a member of the body is: one that is gone, or that may now be absent, breaks a client that relied on it, as a client relies on the Location of a redirect. Header names are compared without regard to case, and a header's schema is compared as a response schema, so its value may only narrow.

Security is compared by what a client presents. An operation accepts any one of its requirements, and each requirement asks for every scheme it names with every scope it lists. The requirements were tightened when a client holding exactly what one old requirement asks for is no longer accepted by any new one: a route that declared nothing and now declares WithSecurity, a dropped alternative, or a new scope. A scheme is identified by what it asks for, its type, where the credential goes and under which name, so renaming a scheme in SecuritySchemes changes nothing, while moving an API key from X-Key to X-Token breaks every client configured for it.

Item and ItemInput

A type used both as a request and as a response is described twice: the component keeps the response's shape, and the request refers to a copy named ItemInput (OpenAPI explains why). The first time a response starts using a type that only requests used, the request body's reference moves from Item to ItemInput. The comparison follows references into components, so it compares the two schemas and finds them the same, and reports only a compatible request-schema-renamed. Any component renamed without changing shape is treated the same way.

Deprecate, then remove

Removing an operation is breaking. Removing one the baseline already marked deprecated is possibly breaking: clients were told to move, and the change is logged rather than failed. That makes the usual sequence two releases long:

// Release 1: still served, marked deprecated.
r.Get("/items/{id}/legacy", handlers.LegacyItem, muzak.Deprecated())

// Release 2: removed. The baseline from release 1 says it was deprecated.

The same holds for a member of a schema marked deprecated.

Versions

Under VersioningURI each version is its own path, so adding /v2/cats beside /v1/cats is a compatible path-added, and dropping /v1/cats later is a breaking path-removed whose message names the version still served:

/paths/~1v1~1cats: The path "/v1/cats" is no longer served; another version of it is served at "/v2/cats". [path-removed]

Under the header-based schemes only one version of an operation reaches the document, as OpenAPI explains, so only that version is compared.

Where a change is located

Location is a JSON pointer into the older document, with ~1 for a slash in a path:

A change inside a named schema is located in components and reported once, however many operations reach it. A component used by requests and by responses can therefore carry two changes at one location, one per direction, with their own severities.

Comparing documents from Go

AssertCompatible is built from three functions you can call yourself, for a command that checks a pull request or compares two deployed services. muzak diff old.json new.json is that command already written, with -fail-on breaking|possibly|never to choose what fails it (see Command Line). The program below shows the three functions at work.

// Command apicheck compares two OpenAPI documents and exits non-zero when
// the second would break a client of the first.
package main

import (
    "log"
    "os"

    "muzak.dev/framework"
)

func main() {
    before := read(os.Args[1])
    after := read(os.Args[2])

    changes := muzak.CompareDocuments(before, after)
    if err := muzak.WriteChanges(os.Stdout, changes); err != nil {
        log.Fatal(err)
    }
    for _, change := range changes {
        if change.Severity == muzak.Breaking {
            os.Exit(1)
        }
    }
}

func read(path string) *muzak.Document {
    f, err := os.Open(path)
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()
    doc, err := muzak.ReadDocument(f)
    if err != nil {
        log.Fatal(err)
    }
    return doc
}

An APIChange carries Severity (muzak.Breaking, muzak.PossiblyBreaking or muzak.Compatible), a Kind from the tables above that never changes once released, a Location and a one-sentence Message. The list is sorted, so the same two documents always give the same output.

What is not compared

Prose is not compared: summaries, descriptions, titles, contact details, categories and tag descriptions change nothing a client can observe, and reporting them would bury the one change that matters. A component no operation reaches is not compared either, since it is not part of the API.

ReadDocument reads the document Muzak writes and refuses anything else with an error that says what and where, rather than comparing a document it only half understood: a document that is not OpenAPI 3.1, a member Document has no field for (an extension of your own, or a keyword such as oneOf that Muzak never writes), a duplicate member, invalid UTF-8, a type JSON Schema does not define, a schema, response or header that is null where one is needed, and a $ref to anything but a schema in the document's own components. Nothing is ever fetched, so a reference to another file or a URL is refused rather than followed. The names an error quotes from the document have every character a terminal would act on escaped, since a failing test prints the error.

Limits

A baseline may come from anywhere, so reading and comparing one is bounded.

Each pair of components is compared once per direction, however many operations share it and however deeply it refers to itself, so an honest comparison spends a small constant per node and finishes in time linear in the size of the two documents. The steps pay for everything a schema holds, not only for visiting it: reading a schema costs a step, one more for every 16 values, members, required names or alternatives it lists, and one for every 256 bytes of text in them, each time it is read, and a node of the documents is weighed the same way. A schema of a few hundred enum values shared by many fields stays well inside the bound, and one holding megabytes cannot be read again at every position for free. Only a document built to be expensive, such as one component compared against thousands of different ones, reaches a bound. The comparison then stops, lists only what it finished comparing, and adds a breaking comparison-incomplete change, so a check fails rather than passing on half an answer.

WriteChanges escapes control characters, invalid UTF-8 and other unprintable characters in the names it prints, so a hostile document cannot write terminal escape sequences into a CI log.

Where to go next

Command Line covers muzak diff, which runs the same comparison on two documents and exits non-zero for a CI step to gate on. OpenAPI covers what the document describes and why a request schema is closed, Versioning covers serving a breaking change beside the old one, and Testing covers the rest of the test client.