Waleed Chayeb.

· Translated from the Arabic

When the fix is published, the race begins

The Procession of the Trojan Horse into Troy, Giovanni Domenico Tiepolo, c. 1760
The Procession of the Trojan Horse into Troy, Giovanni Domenico Tiepolo, c. 1760

Every security update carries two messages at once. The first is for whoever updates: the hole is closed. The second is for whoever waits: the hole is now known, and its description is published in the fix itself. From that moment a race begins between two teams, one installing the update, and one reading it to learn where the flaw was and then searching the internet for every server still on the old version.

This essay is about that race: why it got faster, how our team runs it every day at Levant Host and on the infrastructure that carries Sanarise and its products, and what I advise every company that runs a system or a website to do.

The race got faster

Attackers follow software releases as regularly as developers do. When a fix comes out they compare the new version with the old one and the location of the hole shows itself. Then they write a tool that exploits it and scan the internet for old versions. The tool looks for a version number, so the attack reaches everyone who was late, a large company and a small website alike.

In an analysis published by Mandiant, part of Google, the average time between a fix and the exploitation of its vulnerability was 32 days in 2021 and 2022, and five days in 2023. Artificial intelligence is compressing that further. Reading a fix and writing the exploit used to take an expert and days of their time. Now it takes hours and less expertise.

The window between a security update and the exploitation of its vulnerability

2021 and 202232 days
20235 days
Average time in days; the two lines are drawn to the same scale. Source: Mandiant analysis.

A window once measured in weeks is now measured in days, and for serious vulnerabilities in hours. A company that updates its systems once every three months gives the attacker whole weeks after every fix.

Updating is a daily habit

When the window is measured in days, the race is decided before it starts. A team that waits for the update to come out before asking where this software runs, who updates it, and whether anything will stop working afterwards spends its first days looking for answers, and those are the days the attacker is working.

So our team treats updating as a daily habit that is ready before any news comes out. The habit rests on three pillars: an accurate record of what we have, a clear rule for what is installed at once and what waits, and a way of working that makes the update itself a safe procedure.

The first pillar: knowing what we run. That means every server, every system on it, and every library that an application we wrote depends on. Whoever knows what they have knows within minutes whether the news concerns them.

The second pillar: separating two kinds of release. A security fix is installed as fast as possible, after a test in a staging environment. An ordinary new version waits a few days, because attackers now target the libraries themselves: in March 2024 a planted backdoor was discovered in the xz compression library, days before its infected version would have reached the stable releases of Linux systems. A short wait gives the technical community time to catch something like that.

Security update

  1. Update released
  2. Test in staging
  3. Install

Ordinary release

  1. Version released
  2. Wait a few days
  3. Test in staging
  4. Install
Two tracks for two kinds of release: the security update is installed as fast as possible, the ordinary one waits.

The third pillar: making the update easy. Automated tests and a staging environment turn installing a fix into routine work. A team that fears an update will break its system postpones it, and postponement is what the attacker is waiting for.

One page for all the servers

Our team at Levant Host built an internal tool for this work, and we named it Birdeye. It gathers the state of every one of our servers on one page that the operations team reads every day, with a verdict in one colour at the top: green, amber or red. Under the verdict comes whatever needs a person's eyes, and with each item what to do about it. The most serious cases reach the team within the hour, without waiting for the page.

The tool rests on four rules:

RuleHow we apply it
Every change is an alertWhen a user account is created, a login key is added, a service starts running or a network port opens, the tool compares the server's current state with its previous state and shows the differences, and the team decides whether the change was authorised.
Watching from outsideWe check our servers the way an attacker sees them from the internet: open ports, certificate expiry dates, and the reputation of the mail servers. A server whose regular report is late is an alert in itself and gets checked.
Security news matched to what we useThe tool follows security news and alert sources and brings forward every item that names software we use, so it reaches the team the same day, and we review our versions and update them.
A written response plan for emergenciesThe plan starts with preserving the digital evidence before anything is cleaned, then isolating the affected server, then finding the extent of the breach, then rebuilding, changing every access key and password, and telling the affected clients.

A decision written on a quiet day is more precise than one improvised under pressure.

We designed the tool itself with the same caution. Its permission on any server is limited to reading that server's report, so the servers stay closed even if the tool falls into a stranger's hands. Every day it also verifies that the backups completed and covered every system. Blocking systems run on the servers around the clock, and the watch completes their work: whatever gets past the blocking is seen the same day.

Why we do this ourselves

With us, the person who reads the page is the person who runs the server. The distance between seeing a problem and fixing it is one decision, taken by a team that knows every server by its name and its history. That speed is what a race decided in hours requires.

And the infrastructure we guard is the one we live on. Sanarise's own systems, the WorkLink platform and the Munaweb app run on it, and so do the websites and mail of Levant Host's clients. We guard it with the care an owner gives their own house.

When a major security fix is released for a system our clients use, we write to them: what was released, whom it concerns, and what they need to do. A client who manages their own site is a partner in this race, and updating their plugins and libraries is part of everyone's safety.

What I advise every company

Know what you run. Write a list of every system, every plugin and every library you depend on. That list is what you go back to when the news comes out.

Follow the security bulletins for everything on the list, and install security fixes within days of their release. Stay on versions that still receive fixes; a version whose support has ended keeps its holes open for good.

And ask whoever hosts your site and whoever writes your software two questions: how would you know that one of your servers had been entered? And when did you install your last security fix? A clear, quick answer is the sign of a team that is watching.

The race repeats with every fix that is published. Whoever has made updating a daily habit enters it ahead every time.