The green check that never looked inside the Bundle

In a public repository of integration test fixtures — HL7 v2 messages and the FHIR Bundles their conversion produces, maintained by an NHS genomics service — there is a value we read twice without seeing it:

"endpoint": "https//fhir.mft.nhs.uk/Endpoint/cymru.nhs.uk"

The colon after the scheme is missing. Two lines further down, in the same file, the source endpoint is spelled correctly with https://. That proximity is precisely what makes the defect invisible: the eye compares two strings that look alike and files them as identical.

A validator caught it; careful rereading did not. And the interesting question is not how it got there, but why none of the automated checks that file already passed through had flagged it.

Why the check that existed could not catch it

MessageHeader.destination.endpoint in FHIR R4 is mandatory — cardinality 1..1 — and typed url: the specification defines it as a Uniform Resource Locator, an address reached directly using the stated protocol. Without the colon there is no scheme, and the string stops being an absolute address: it becomes a relative path. A message routed there arrives nowhere.

Here is the part that matters to anyone evaluating a validation tool. The R4 definition of the url type carries a regular expression, and that expression is \S* — any run of non-whitespace characters. We downloaded the StructureDefinition to be sure, because this was exactly the kind of detail we did not want to take on trust. A validator checking the primitive type as the specification defines it will therefore accept that value, and it is right to: it is applying precisely the rule it was given. The constraint this string breaks is not written in a form a machine can enforce on its own.

So a file can clear every available check and still carry an address that resolves to nothing. That is not a flaw in the validator. It is the gap between conformant and working, and in hospital integration that gap is where the expensive failures live — the ones that surface on the first day of real traffic and not a minute earlier.

The second case is ours

A few days earlier we had run a FHIR Bundle with thirty-three entries through our own validator. The answer came back: zero errors, zero warnings. It looked like good news. It was the worst possible symptom. The tool had never looked inside. It validated the wrapper — the Bundle resource, its structure, its top-level fields — and stopped there, never descending into entry[].resource. Thirty-three resources unexamined, and a green tick.

We found it by running the exercise on somebody else’s files, fixed it the same day, and said so plainly in the chat.fhir.org discussion where it came up. Result paths now report entry[N].resource, and a profile declared in meta.profile but not installed produces an explicit warning instead of a clean result.

That last point deserves a sentence of its own, because the confusion is widespread. In R4, meta.profile is the list of profiles a resource claims to conform to: an assertion made by whoever produced the file, not a finding. Claiming conformance and having it verified are two different facts, and an honest tool should always make clear which of the two it established.

The third case: tests that cannot fail

The same pattern reappears in round-trip tests — convert a v2 message to FHIR, convert it back to v2, compare the result with where you started. The common minimal check is that what comes back is a v2 message at all, that is, that it begins with MSH. In the repository we started from, the authors themselves document this step as a sanity check rather than validation, which is a piece of honesty worth more than it usually gets.

The trouble starts when nobody writes that down. A check like this catches a crash, not a silent loss. You can drop a patient identifier, a field inside a segment, an entire OBX carrying a result — and the rebuilt message will still begin with MSH. The test stays green indefinitely, and the longer it stays green the more firmly everyone believes that conversion is covered.

The difference between a liveness check and a real oracle is exactly this: a useful test has to be able to fail for the reason you wrote it. The step to get there is short and changes everything. Diff the round-tripped message against the original, and measure that difference against a declared loss set versioned alongside the test case. Expected loss: pass. Loss outside the set: fail. A set that no longer matches the observed difference: the case has drifted and needs a human to look at it. As a side effect, that set is also the document answering “what could not be mapped, and why” — a question that normally gets answered in a wiki page, where it quietly goes stale.

Three questions to ask next time

The three stories above are the same story: a check that passes because it was never in a position to fail. The next time a validation report says everything is fine — whether it comes from a vendor, a tool, or your own team — three questions change the conversation.

1. What did it validate against? Not “against FHIR”: which profiles, from which packages, at which versions. A missing package or a different version changes the outcome, and two tools can both report “conformant” having looked at entirely different things.
2. How deep did it go? A result on a Bundle that never names the individual entries is not a result on that Bundle.
3. What can it tell you it did not check? This is the question that separates a useful tool from a reassuring one. An explicit refusal to start beats an ambiguous success every time.

In our experience this is the least visible risk in an integration project, because it does not present as an error. It presents as a row of green lights nobody has any reason to question — right up until real messages start flowing. At that point it becomes very visible indeed.

The endpoint defect was found by BridgeLab, the workbench we use to inspect and validate v2 messages and FHIR Bundles. The defect in our own validator was found by the exercise of pointing it at somebody else’s files — the kind of evidence no internal test suite substitutes for.

Sources

• HL7 FHIR R4, Data Types and MessageHeader — v4.0.1, Technical Correction #1, 1 November 2019
• HL7 FHIR R4, Resource (definition of meta.profile) — v4.0.1
• Public repository of the fixtures referenced: nw-gmsa/Testing
• Discussion: Shared corpus of hard v2 → FHIR test cases, #v2 to FHIR channel on chat.fhir.org