Managing releases

Written for: Administrator

A new version of PHP, of the agent, or of the panel itself does not go to every server at once. It walks: a soak on the beta channel, then a canary machine, then the fleet in waves, with a wait between each pair and a health check that can stop the whole thing. This page is how you drive that from the panel, and how to read what it tells you.

Only administrators see this. Resellers and end customers never see which versions exist, let alone which machine is on which one.

Open Servers → Releases.

The three questions the board answers

The page is laid out in the order you actually ask them.

1. May a wave start right now? The banner at the top. It is green when the door is open and amber when it is not, and when it is not it says why in a full sentence — a Saturday, a public holiday, or an incident somebody opened.

2. What is waiting on me? The pipeline. Every version with a badge for its risk class, its channel, and how far its soak has run.

3. Which machines would it touch first? The wave groups, at the bottom. Drag a server from one card to another to change the order the fleet is updated in.

Risk classes

The badge on every release decides everything about how it travels.

minor-runtime — a PHP patch, a LiteSpeed build. Nothing on the node keeps state about it, so a downgrade is a real answer, and the panel promotes it by itself once its soak comes up clean.
platform — the agent, or the panel. It promotes only when a person clicks, its canary is at least three machines including a busy one, and the panel rolls forward rather than back: a database that has migrated does not un-migrate because a binary went backwards.
security-critical — a fix that cannot wait. The timers compress. Nothing else does.

An unknown package is treated as platform, which is the conservative answer. You can override the class of a package under Advanced → risk classes if the name-based rule gets it wrong.

Approving a release

A release whose class needs a person appears with the state awaiting_approval, and the banner at the top of the page counts them. Open it to see the evidence:


Soak	how many days it has been on beta, against how many its class asks for
Canary	how many canary machines took it, and how many passed their gate
Error logs	the rate customers' error logs grew, as a multiple of before
Exposure	how many synthetic requests really succeeded on it

These are the same four numbers the panel promotes a minor-runtime release on. There is no second, hidden set.

If the soak is clean, Approve moves it to the next channel. If it is not, the button reads Approve anyway and requires a reason — which goes into the promotion ledger with your name on it, and stays there.

Reject also needs a reason. It is the only record of why a release did not go out.

Wave groups

test is on beta and holds the operator machines with synthetic customer sites on them. wave-1, wave-2 and wave-3 are on stable, in blast-radius order.

Two things the panel insists on:

Wave 1 needs a genuinely busy server. The card shows how many hosting accounts each machine carries and marks the busy ones. A first wave of idle machines exercises none of the code paths the release will actually meet, so the board warns when wave 1 has none — and a rollout planned that way is refused outright.
A new server lands in a late wave by itself, round-robin, as soon as you open this page. Drag it forward when you know what it carries.

A group can hold itself back with a channel pin or a version ceiling — a customer who wants to stay a release behind. Every class honours that except security-critical, which reaches them anyway unless you opt the group out explicitly, and that opt-out is recorded with your name and your reason.

What stops a rollout

While a wave runs, each machine is checked after its own update — not "did apt exit zero", but whether its customers are as well served as they were ten minutes earlier: the websites still answer, their error logs are not filling up, mail is not queueing.

When that check goes red the panel does four things and then stops:

rolls that machine back to the version it had,
halts the rollout, leaving the machines it never reached marked as skipped,
opens an incident, which blocks every further wave until you close it,
sends the paging mail.

It never retries. A rollout that retried a failed machine would flap, and every cycle of that is another outage for the sites on it. The fleet stops where it stands and waits for you.

Blackout windows

No wave starts on a Friday, Saturday or Sunday, on a Dutch public holiday, or while an incident is open. This is not superstition: a wave bakes for hours and the whole value of a bake is that somebody is watching it.

Trying anyway is refused, with the window named. The only way past it is the emergency lane.

The emergency lane

The red block on a release's panel. It asks for a reason before it does anything, and it lists what it is about to override:

the beta soak — hours instead of days,
the bake between waves — minutes instead of a day,
a group's channel pin and version ceiling,
blackout windows,
the softer health checks.

And what it does not override, which is the part worth trusting:

the canary. No class may skip it, and there is no path in the panel that plans a rollout without one.
a red gate. The rollout still halts, still rolls the failing machine back, and still pages.

Everything the lane does is written down: on the release, on the rollout, and in the audit log.

From a terminal

Everything above is the API, and the API is the same one the screen uses.

# What is in the pipeline
$ curl -s -b cookies https://panel1.corecp.dev/api/v1/releases |
    python3 -c 'import json,sys
for r in json.load(sys.stdin):
    print(r["package"], r["version"], r["risk_class"], r["state"], r["channel"])'
corecp-php85 8.5.3 minor-runtime approved stable
corecp-agent 0.34.1 platform awaiting_approval beta

# Run the promotion policy now instead of waiting a quarter of an hour
$ curl -s -X POST -b cookies https://panel1.corecp.dev/api/v1/releases/tick

# May a wave start?
$ curl -s -b cookies https://panel1.corecp.dev/api/v1/releases/windows |
    python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["open"])'
True

On a server itself, the same health check the gate uses is one command:

# root@stck1
$ corectl release gate

It exits 0 when the machine is no worse than its baseline and 1 when it is, naming the site or the service that changed.

Everything on this page is in your language

The release board explains a lot: why a window is closed, what a risk class means, who may promote a version, why a soak is not clean yet. Those sentences are composed on the server, and until round 2's final pass they came out in English even when your panel was set to Dutch.

That is fixed, in a way you can see: switch the panel to Nederlands at the top right and the same explanations appear in Dutch, without the page reloading. The server no longer sends a sentence — it sends a reference plus the numbers that belong in it, and the panel builds the sentence.

Nothing changes on the command line, where the English text is still what you get:

ssh panel1.corecp.dev 'corecp-panel releases policy --config /etc/corecp-panel/panel.yaml'
ssh panel1.corecp.dev 'corecp-panel waves board --config /etc/corecp-panel/panel.yaml'

If you do still see an English sentence on this screen with the panel set to Dutch, report it: that is a hole in the translation list, not a decision.