Release quality gates: what to actually gate on

A quality gate is a promise you make to yourself in advance: if this is not true, we do not ship. Almost everybody who introduces one is surprised by how quickly it stops working. Within a quarter it is either being overridden as routine, or it has been loosened until it never fires. Both outcomes look like the gate failing. Usually the gate was wrong to begin with.

Why gates get switched off

There are only a few reasons, and they repeat across teams. The gate blocked something that was obviously fine, so people learned that overriding it is normal. It was set on a number nobody agreed to, so the first person to hit it argued with the number rather than the problem. It fired at a moment when nothing could be done about it, which teaches everyone to route around it. Or it was one undifferentiated wall, so a trivial issue and a serious one both produced the same red light, and the red light stopped meaning anything.

Notice that none of these is the gate being too strict. Strictness is not the failure mode. Being uninformative is.

Gate on the things that would change the decision

The test for whether something belongs in a gate is simple: if this were false, would we genuinely hold the release? If the honest answer is no, it is a metric worth watching, not a gate. Putting it in the gate anyway is how you train the team to ignore all of it.

In practice the things that pass that test cluster into a handful of areas:

• How much passed, as a rate you agreed in advance
• Whether anything failed at a severity you consider unacceptable, stated separately, because one bad failure disappears inside a good percentage
• Whether known defects are still open above a severity you choose
• Whether the work you planned to do was actually finished rather than merely started
• Whether the things this release changes are covered at all
• Whether anything is disabled, skipped or quarantined without somebody knowing
• Whether the evidence is recent enough to describe the code you are shipping

That last one catches more teams than expected. A pass from three weeks and two merges ago is not evidence about today's build, and a gate that accepts it is measuring history.

The numbers belong to you, not to the tool

Any tool that ships a fixed pass-rate threshold is guessing about your product. Ninety per cent might be reckless for payments and unreachably strict for an internal admin screen with a flaky third-party dependency. The number has to come from the team that will have to live with it, and it has to be agreed before the release it judges, or it is not a standard, it is an argument you are having under pressure.

It also has to be allowed to differ. A gate that is identical for a hotfix and a quarterly release will be wrong for one of them. Set a sensible baseline for the project and let a particular release adjust it deliberately, on the record, rather than by everyone quietly agreeing to ignore it this once.

Split the wall into must and should

This is the single change that makes gates survive contact with a real release. Some conditions genuinely stop a ship. Others are real problems you might still accept, knowingly, because the fix is scheduled and the exposure is small.

Mark each one. A must cannot be waived. A should can, but only out loud, by a named person, with a reason recorded at the time. Without that split you get the two failure modes at once: the gate is too strict to respect and too blunt to inform, so it gets overridden, and the override leaves no trace of what was accepted.

Put it where something can still be done

A gate that fires at the release meeting, when the deadline is tomorrow, is not really a gate. It is a report. Most of these conditions can be evaluated continuously, so the team finds out on the Tuesday that coverage on the new stories is thin, rather than on the Friday when the only options are ship anyway or slip.

Keep the formal check at the release point, because that is when somebody has to decide. But if that is the first time anyone sees the state, the gate has already failed at the useful part of its job.

A gate should say which one, and why

When a gate fails it should name the condition, show the number against the bar, and point at the evidence. Not ready, on its own, tells nobody what to do next, and a gate that produces an argument rather than an action will not last.

AxonQA ships ten conditions covering the areas above. The thresholds are yours: you set the numbers on the project, mark each condition as must or should, and adjust the set for a particular release when there is a reason to. Conditions that do not apply, such as API pass rate on a release with no API checks, drop out rather than being scored as empty passes. You can see the full set, and a real assessment made against it, on our release readiness page.