When Declarative Firewall Sync Prunes Rules and Cuts Access

A declarative sync loop that reconciled firewall rules against a source of truth in code did exactly what it was told, and that was the problem. One afternoon it removed a set of live rules and cut off traffic that had been working for months. Nobody had changed the code. The rules it deleted had been added by hand in a console, outside the declarative source, and the sync treated anything not in the source as garbage to be pruned. This is a postmortem of that incident, written generic on purpose, and the design changes that keep automated prune from becoming a weapon.

What happened

The environment had a reconciler. A declarative source in version control described the desired firewall and security-group rules for each network boundary. On a schedule, an automation read that source, compared it against the live rule set on the infrastructure, and made the live set match. New rules in the source got created. Rules present live but absent from the source got deleted. That delete-to-match behavior is the whole point of declarative reconciliation, and it is also the entire cause of the outage.

Weeks earlier, someone had added a few rules directly in the infrastructure console to unblock an urgent need. A specific outbound path and a couple of ports that a service depended on. Those rules were never written back into the declarative source. They lived only on the live infrastructure. For a while nothing forced the question, because the sync had not run against that boundary since the manual edit, or had run in a mode that did not prune.

Then a normal sync ran. It saw rules on the live side that did not appear in the declarative source. By its contract, those rules were drift to be corrected, so it deleted them. The outbound path went away. Health checks that traversed one of the pruned ports began to fail. Traffic that had depended on the hand-added rules stopped. The automation reported success, because from its point of view the live state now matched the source perfectly. The alert that mattered did not come from the sync. It came from the services that could no longer reach what they needed.

Timeline in brief

The sequence is short and it repeats across many teams, which is why it is worth stating plainly.

1. A rule set is defined declaratively and reconciled automatically. Correct so far.
2. An urgent need arrives. Someone adds rules by hand in the console to fix it now. The declarative source is not updated, so a second source of truth quietly comes into existence.
3. Time passes. The manual rules keep working. The gap between live state and declarative source widens with nobody watching it.
4. A routine sync runs with prune enabled. It sees the manual rules as unmanaged drift and deletes them.
5. Access breaks. The failure surfaces downstream, not in the tool that caused it, so the first minutes of the incident are spent looking in the wrong place.

The recovery itself was fast once the cause was clear. The lost rules were re-added, this time into the declarative source, and access came back. The uncomfortable part was not the fix. It was realizing the automation had been one scheduled run away from this outcome for weeks, and that the design made the outcome inevitable rather than unlucky.

Root cause: two sources of truth for one rule set

The headline cause was not a bug in the sync. The sync worked as designed. The cause was that two places could both define reality, and only one of them was allowed to survive.

The declarative source claimed to be the single source of truth. The console let humans write rules directly, which made the console a second source of truth in practice. As long as both were used, the system had two authorities that disagreed, and the reconciler was built to resolve every disagreement in favor of the source by deleting whatever the source did not contain. A manual rule was not an addition the system would preserve. It was a difference the system was designed to erase.

Three specific decisions turned that structural flaw into an outage.

First, the automation enforced the source as the only truth by pruning, while a path for manual edits stayed open. You cannot have it both ways. If the source is authoritative and prune is on, manual edits are not shortcuts, they are time bombs with an unknown fuse.

Second, the sync applied changes immediately with no diff preview and no approval. It never showed a human the list of rules it was about to delete. A dry run that printed "about to remove 3 rules including one outbound path" would have stopped the incident in the time it takes to read one line.

Third, and most important, the design treated removals and additions as the same kind of change. Adding a rule that the source contains but the infrastructure lacks is low risk. The worst case is that traffic which was already blocked stays blocked a moment longer. Removing a rule is the dangerous direction, because it can cut access that is actively in use, and the blast radius is only known after the fact. Giving both operations the same automatic, unguarded treatment meant the safe operation and the destructive one ran with identical trust.

Root causes and countermeasures

The fixes map one to one onto the causes, and each one narrows the failure independently, so a gap in one does not reopen the outage.

Prevention: make prune the guarded operation

The guiding principle is that a declarative system is only as safe as its treatment of deletion. Creating is forgiving. Deleting is not. The redesign made removal the operation that has to earn its way through a gate, while leaving the safe majority of changes automatic.

Prune now runs in two stages. The first stage computes the difference between the source and the live state and prints it, grouped by direction. Additions on one side, removals on the other. It applies the additions on its own, because an addition cannot cut existing access. It does not apply a single removal. Instead it stops and presents the removals for approval, with enough context to judge each one. A human reads the list and approves, or does not. Only after explicit approval does the removal stage run.

That asymmetry is the core of the fix. In steady state, most syncs contain only additions or no changes at all, and those run untouched with no human in the loop. The rare sync that wants to delete something is the rare sync that gets a human in the loop. The cost of the gate lands only on the operation that can cause an outage, which is exactly where you want the friction.

A threshold alert backs this up. If a proposed prune would remove more than a handful of rules at once, that is treated as a signal that something is wrong with the input, not as a large but routine change. A sync that suddenly wants to delete twenty rules is far more likely to be reading a broken or empty source than to reflect a real intent to remove twenty rules. The alert fires before the diff is ever approved.

Prevention: single source of truth and a protect list

The gate on prune limits the damage of drift. Eliminating drift is the other half. If only one place can define rules, the reconciler never sees a legitimate live rule as garbage, because every legitimate rule lives in the source it reconciles against.

That meant closing the console path. Direct edits to firewall and security-group rules through the console were disabled for humans, so the declarative source became the single source of truth in fact and not just in intention. Urgent changes still happen, but they happen as a small, fast change to the source that flows through the same pipeline, which is only slightly slower than a console edit and leaves the two authorities in agreement instead of in conflict. A separate drift detector runs on a schedule and alerts when live state and source disagree at all, so a gap is caught as a warning long before a prune could act on it.

A protect list closes the last hole. Some rules must never be pruned no matter what the source says, because they are the rules that keep the operators reachable. Management access and the paths that health checks travel are the obvious examples. Those are marked as protected and excluded from reconciliation entirely. Even a completely empty or corrupted source cannot delete them, which means the failure mode where a bad source locks everyone out of the very infrastructure they need to fix it is off the table. The protect list is short by design, and every entry earns its place by answering one question: if this rule vanished, could we still get in to repair the damage.

Here is the shape of a reconciler config that carries these guarantees. The exact schema is not important, the properties are.

reconcile:
  # Additions apply on their own. Removals never do.
  auto_apply:
    additions: true
    removals: false
  # Removals produce a diff and wait for a human.
  prune:
    require_approval: true
    dry_run_first: true
    # A prune larger than this is treated as a bad input, not a big change.
    alert_threshold: 5
  # Rules the reconciler is forbidden to delete, whatever the source says.
  protect:
    - name: management-access
      match: { port: 22, direction: inbound }
    - name: health-check
      match: { port: 8080, direction: inbound }
  # Snapshot before any change so a bad apply rolls back in one step.
  snapshot:
    before_apply: true
    retain: 10

Prevention: atomicity, snapshots, and observability

The final layer assumes a bad change gets through anyway, because eventually one will, and asks how fast you can undo it. Before any apply, the reconciler takes a snapshot of the current live rule set. If an apply produces a broken state, recovery is restoring the snapshot rather than reconstructing rules from memory under pressure. Changes apply atomically where the platform allows it, so a run does not leave half the rules updated and half not, which is its own kind of outage that is harder to reason about than a clean failure.

Observability closes the loop. The most useful single signal turned out to be the crudest one: the count of active rules per boundary, alerted when it drops sharply. A rule count that falls from forty to thirty-four in one sync is a question worth asking immediately, and it does not depend on understanding what any individual rule does. Pair that with the drift detector, which watches for live state and source disagreeing, and the two together catch both the slow problem (drift accumulating) and the fast one (a sync deleting in bulk) well before a human would notice traffic failing.

The general lesson: declarative is not the same as safe

The comforting story about declarative automation is that it removes human error. Write the desired state, let the machine enforce it, and stop making manual mistakes. That story is half true. Declarative reconciliation does remove a class of drift, and for additions it is genuinely safer than hand edits. What it does not do is make deletion safe, and it quietly raises the stakes of having more than one source of truth.

The moment two places can define reality, an enforcer that resolves every conflict by deleting is not a safety mechanism. It is an automated way to act on the wrong truth at machine speed. The failure here was not that automation was used. It was that a destructive operation was granted the same trust as a safe one, in a system where a second, informal source of truth was allowed to exist.

The rule that comes out of it is short. Keep exactly one source of truth, and detect drift from it rather than letting drift form. Let automation apply the safe direction freely. Put a human gate on the dangerous direction, because deletion is the operation that cuts access, and no amount of declarative tidiness changes that. A pipeline that prunes without a human in front of the prune is not more reliable than a careful operator. It is a careful operator's worst afternoon, scheduled to run on a timer.