How to tell whether a release is ready to ship

Somebody has to say yes. Usually it happens near the end of a call, after a run-through of what got tested, and it comes down to whether the people in the room feel alright about it. Nobody writes down what the yes was based on, because at the time it seems obvious. Three weeks later, when something has gone wrong in production, it is not obvious at all, and the only honest answer to what did we know is that nobody wrote it down.

The fix is not more testing. It is deciding, before you are under pressure, what would have to be true for this release to be safe. That is a short list, it is boring to write, and it changes the meeting completely.

All tests passing is not the same as ready

A green suite answers a narrower question than people think. It tells you that the checks you happened to write, on the areas somebody thought to cover, passed on the code that was running when they ran. It says nothing about the story that shipped without a test, the check that has been quarantined since March, or the high-severity bug still sitting open in the tracker.

This is why pass rate on its own is a poor gate. One catastrophic failure among three hundred passes still reads as ninety-nine per cent. If the only number you look at is the percentage, the worst failure you could have is invisible in it.

Criteria worth setting

A useful set is short enough to hold in your head and specific enough that two people would score it the same way. These are the ones that earn their place in most teams:

• Pass rate across everything in the release, against a number you set in advance rather than after seeing the result
• No failure at or above a priority you choose, stated separately from pass rate so one severe failure cannot hide inside a good percentage
• No open bug above a severity you choose, counted against bugs actually linked to this release
• Every manual phase finished rather than merely started, because a phase nobody ran is not a phase that passed
• Coverage of the stories this release touches, not the whole product, since the rest is not what you are shipping today
• Nothing quarantined or skipped without somebody knowing, because a disabled test is an unknown wearing a green tick
• Evidence recent enough to mean anything, so a pass from three weeks and two merges ago does not count as current

Split them into must and should

Not every criterion carries the same weight, and pretending otherwise is what makes gates get ignored. Some things genuinely stop a release. Others are real problems you might still ship with, knowingly, because the alternative is worse.

So mark each one. A must cannot be waived: if it fails, the release is not ready, and that is the end of the conversation. A should can be waived, but only out loud, by a named person, with a reason recorded at the time. That distinction is the whole value of the exercise. Without it you get one of two failure modes: a gate so strict that everyone learns to override it, or a gate so soft it never says no.

The answer has three shapes, not two

Most teams model this as a straight yes or no, and then discover that real releases do not divide that way. The common case is that everything critical holds, one or two lesser things do not, and you ship anyway because the fix is scheduled and the exposure is small. That is not a no-go. It is also not a clean yes, and calling it one throws away the only record of what you accepted.

So allow three outcomes. Ready, meaning everything you set holds. Not ready, meaning something that must hold does not. And ready with accepted risk, meaning the musts hold, some shoulds do not, and a person has put their name to each one with a reason. The third is the honest description of most releases, and it is the one a binary gate cannot express.

Write it down before you need it

The criteria have to exist before the release they judge. Set after the results are in, they are not criteria, they are a justification, and everybody in the room knows it. Agree them at the start of a cycle when nobody is under pressure and the answer is not yet inconvenient.

Then record the decision with the criteria attached, as they stood on the day. A dashboard that recalculates is useful for knowing where you are right now, and useless for the question you will actually be asked later, which is what you believed when you shipped. Those are different questions and they need different artifacts.

What good looks like afterwards

You know this is working when the release meeting gets shorter and duller. The discussion moves from whether it feels alright to whether the two outstanding items are worth accepting, which is a decision a group can actually make. And when something does go wrong, the conversation starts from what was known and accepted, rather than from whose fault it was.

AxonQA was built around this. You set the conditions on a project, mark each as must or should, and the evaluator answers with one of those three verdicts from your test cases, automation runs, API checks, open bugs and manual phases together. Each assessment is written once and never rewritten, with the conditions and evidence stored on it as they stood, so the record still means something a month later. You can read a real one on our sample report without signing up for anything.