Silent Failures on the Headless Seam

Three production defects on the same boundary, in the same release, none of which produced an error message. What decoupling actually costs, why the usual pipeline cannot see it, and the three artifacts that made the boundary checkable.

The seam nobody owns

Decoupling a CMS from its front end splits one system into two, each with its own repository, tests, deploy pipeline, and on-call story. The split is worth it. What it creates, and what almost nobody budgets for, is a seam: a contract between the two halves that exists in production and nowhere else.

Inside either half, the tooling is good. Drupal has an opinion about whether Drupal is correct. TypeScript has an opinion about whether the front end is internally consistent. Both are honest. Neither has any opinion about the other.

So the seam has a property the rest of the system does not: it is the only place where both sides can be individually correct and jointly broken. And because no test in either repository is looking at it, the failures do not announce themselves. They wait for a person to notice.

In a single release we shipped fixes for three of them. It is worth walking through all three, because they look unrelated and they are not.

Failure one: the preview that served published content

Editors clicked Preview in Drupal and got the page that was already live. The signed preview URL validated. Draft mode enabled. The route rendered. The banner announcing "you are viewing a draft" appeared above content that was not a draft.

The cause was one loader. The preview route read the draft-mode flag and used it to decide whether an unpublished node should 404 — but never to decide what to fetch. It called the same query the public site calls, which asks Drupal for a node by path and receives the default revision. For a published node, the default revision is the published one. A draft of a published node is a forward revision: newer, non-default, deliberately invisible to anonymous traffic.

The query was behaving exactly as specified. The preview system was complete except for the single capability it existed to provide.

Note what a linter sees here: a variable that is used. Unused code gets flagged. Code used for the wrong purpose looks like working code.

Failure two: the contract that could delete every page at once

Rename a field in Drupal and the front end does not degrade — it stops.

GraphQL validates a query as a whole document before executing any of it. One selected field that no longer exists does not return partial data with a warning; it rejects the entire query. Our node query is a single ten-thousand-character document behind every content page on the site. One stale selection in it takes out every page running it, simultaneously, in production.

It has happened to us twice. Both deploys were green.

The mirror-image failure is quieter still. Add a field in Drupal and the front end simply never sees it. There is no error, no symptom, and no upper bound on how long the drift can sit there. The additive case has no failure mode at all — which is precisely why it accumulates.

Failure three: the access rule that was not where anyone was looking

Our front end's service account carried the administrator role. That reads like a serious finding: a public-facing client authenticating from an admin account.

It was not, and the reason matters. With simple_oauth at role granularity, account roles are stripped from a client_credentials token. The scope is the boundary. We confirmed it with one account and two consumers: same user, roles []; scope previewer returns the unpublished draft, scope nextjs_site returns null. Identical identity, different answer.

So the admin role had been granting nothing. It was inert — and one configuration change away from not being inert. A real defect whose severity was nothing like its appearance, sitting in the gap between the access rules people read in the interface and the access rules the request path actually consults.

The shape they share

Three defects, three subsystems, one shape. In each case the system's degraded behavior was indistinguishable from its correct behavior:

• Preview served a page. Just the wrong revision of it.
• The schema check passed. It was checking each side, not the seam.
• The service account worked. Its stated privileges were fiction.

This is what makes seam bugs expensive out of proportion to their difficulty. Every one was a small fix. Every one survived indefinitely, because surviving is what they do. A system that crashes gets fixed on Tuesday. A system that quietly serves the wrong thing gets fixed whenever somebody happens to look closely, which may be never.

Why the pipeline is structurally blind

It is tempting to call this a testing-discipline problem. It is not. Look at what each layer actually measures:

• TypeScript checks the shape you claim a response has. Queries are strings; tsc has no opinion about their contents.
• The front-end build compiles code. It never contacts the CMS.
• Unit tests mock the client, so they assert against fixtures that were true when written.
• CMS tests pass, because from the CMS's perspective nothing is broken. Renaming a field is legitimate.
• Health checks confirm the service answers. They do not confirm it answers correctly.

Every layer is honest about something adjacent to the question. The seam is not represented in any artifact, so nothing can check it. You cannot test your way to covering a thing that does not exist in your repository.

The fix in all three cases: make the invariant an artifact

The pattern that resolved all three was the same. Take the invariant that lived in someone's head and give it a physical representation something can check.

A contract file. The CMS exports its GraphQL schema and we commit it in the front-end repository — roughly 97KB, 136 types. A unit test builds the schema from that file, captures the query each fetcher would send, and validates it. Breaking drift now fails at pull-request time naming the exact field; additive drift shows up as a diff a human reads. It lives on the front end deliberately: the front end declares what it needs, and whatever CMS sits behind the seam must satisfy it. That is what keeps the contract portable rather than Drupal-shaped.

Worth saying plainly: we wrote almost none of this. drush graphql:dump and drush graphql:detect-breaking-changes already exist upstream, and the second already distinguishes breaking from additive. We also nearly abandoned the approach over a wrong assumption — that disable_introspection would block schema export. It does not. That flag installs a query validation rule; printing the schema walks the type map in-process and executes no query. Check the assumption that would kill the design first.

A resolver that returns what it authorizes. Preview now uses a dedicated field that loads the latest revision — preferring the latest translation-affected revision, so a Spanish draft does not silently resolve to the English working copy — and checks view access on the loaded revision object. There is no separate authorization branch to drift out of sync with the loading logic. The object returned is the object authorized.

A scope per job. Two consumers, two scopes: one that reads published content, one that reads drafts. The public client cannot leak a draft because it holds no scope that can see one. Secrets moved into encrypted, environment-sourced configuration rather than plaintext in exported config, which removes an entire recurrence vector — the class of break where the two halves disagree about a shared secret and the handshake fails for reasons no log explains.

The rule worth stealing

Where a system's degraded mode is to serve something reasonable, the absence of an error proves nothing. Acceptance tests on such a system must be written to detect a success, never the lack of a failure.

Ours for preview is one sentence: open a draft of an already-published node and confirm you see your unpublished edits. Not "confirm preview loads" — that passes while broken. Not "confirm no errors" — there were never any errors. The only question that discriminates is whether the bytes on screen are the ones that are not live yet.

We learned the same lesson the expensive way on the identity side. Our first verification plan said "confirm token minting still succeeds." A blocked service account still mints tokens perfectly well — it is every subsequent request that 401s. The plan would have certified an outage as healthy. Never verify a credential by minting a token; verify it by spending one.

Both mistakes have the same root: measuring the step before the one that matters. Issuance instead of use. Rendering instead of correctness. The measurement is easy to take, which is exactly why it is the one you reach for.

Where this leaves us

These three fixes are one capability wearing three hats: a management plane and a delivery plane that can no longer drift apart without something saying so. The schema contract guards what crosses the seam. Scoped machine identity guards who may cross it. Encrypted, environment-sourced secrets guard the handshake itself. None of it is exotic, and none of it is Drupal-specific — the contract binds to a normalized content model, not to CMS internals, which is the property that lets the same guard sit on a different backend.

The honest summary of the release is not that we fixed three bugs. It is that we found three places where the system had been telling us it was fine, and replaced each one with something that can tell us when it is not.