Rolling Back a Bad Remote Deployment #

Rolling back a federated remote should be a pointer change measured in seconds, not a rebuild measured in minutes — this page sets up the immutability, manifest indirection and cache rules that make that true, and the runbook for when it is not.

Rollback is the property that makes independent deployment safe to do often. If reverting one remote requires a host release or a CDN purge race, teams deploy less, batch changes, and make each release riskier. It builds on versioning strategies for remote apps.

What makes rollback fast or slow Rollback speed is decided entirely by whether previous releases are still addressable and whether the pointer to them is small and short-lived. Mutable entry, no history One URL overwritten on every release Rollback means rebuilding the old commit CDN caches race the new upload Recovery measured in tens of minutes Immutable assets, manifest pointer Every release keeps its own URL Rollback edits one small JSON file Assets are already cached at the edge Recovery measured in seconds
Everything else on this page follows from keeping old versions reachable — that single rule is the whole capability.

Prerequisites #

Step 1 — Make every release independently addressable #

Rollback is only possible if the previous version still exists at a URL. That means version-prefixed paths and a firm rule against deleting them.

https://cdn.example.com/cart/4.2.0/remoteEntry.js     ← still there
https://cdn.example.com/cart/4.2.1/remoteEntry.js     ← still there
https://cdn.example.com/cart/4.3.0/remoteEntry.js     ← current
// Cache-Control per asset type. Only the manifest is short-lived.
const HEADERS = {
  'remoteEntry.js': 'public, max-age=31536000, immutable',
  '*.chunk.js':     'public, max-age=31536000, immutable',
  'remotes.json':   'public, max-age=30, must-revalidate',
};

Exclude these paths from any storage lifecycle rule. A cleanup job that deletes objects older than ninety days silently removes your ability to revert past that point, and it will be discovered during an incident.

Step 2 — Keep a history in the manifest #

Knowing which version to roll back to should not require reading a deploy log under pressure.

{
  "cart": {
    "version": "4.3.0",
    "entry": "https://cdn.example.com/cart/4.3.0/remoteEntry.js",
    "integrity": "sha384-…",
    "history": [
      { "version": "4.2.1", "entry": "https://cdn.example.com/cart/4.2.1/remoteEntry.js", "integrity": "sha384-…", "publishedAt": "2026-07-21T09:14:00Z" },
      { "version": "4.2.0", "entry": "https://cdn.example.com/cart/4.2.0/remoteEntry.js", "integrity": "sha384-…", "publishedAt": "2026-07-14T11:02:00Z" }
    ]
  }
}

Carrying the integrity hash in the history matters. A rollback that drops the hash silently disables the verification described in subresource integrity for remote entry files for that remote, and nobody notices because the page works.

Step 3 — Write the rollback as one command #

What happens between the command and the recovery Because the previous entry is still cached at the edge, the only thing that has to propagate is a small manifest with a thirty-second lifetime. Operator Manifest CDN edge User 1. rollback cart → 4.2.1 2. TTL expires, revalidate 3. next page load 4. manifest names 4.2.1 5. entry already cached, no fetch
The last message is why this is fast: the asset the user needs was never evicted.

Under pressure, a multi-step procedure is a procedure that gets done wrong.

// scripts/rollback.mjs — one remote, one step back, or an explicit version
const [, , remote, target] = process.argv;

const manifest = await readManifest();
const spec = manifest[remote];
if (!spec) throw new Error(`unknown remote: ${remote}`);

const previous = target
  ? spec.history.find((h) => h.version === target)
  : spec.history[0];
if (!previous) throw new Error(`no such version in history: ${target}`);

// The current version moves into history; it is not deleted.
manifest[remote] = {
  ...spec,
  ...previous,
  history: [{ version: spec.version, entry: spec.entry, integrity: spec.integrity }, ...spec.history],
  rolledBackFrom: spec.version,
  rolledBackAt: new Date().toISOString(),
};

await writeManifest(manifest);
console.log(`${remote}: ${spec.version}${previous.version}`);

Recording rolledBackFrom is what stops the next deploy from silently re-shipping the broken version. A pipeline that sees a rollback marker for the version it is about to publish should refuse without an explicit override.

Step 4 — Understand what users actually experience #

A rollback does not reach everyone instantly, and knowing the shape of the transition prevents a second, unnecessary intervention.

Users who load the page after the manifest TTL expires get the previous version immediately. Users mid-session keep running the broken version until they reload — the module is already evaluated and cannot be replaced in place. Users behind a CDN edge that has not yet revalidated the manifest may lag by up to the TTL.

With a 30-second manifest TTL, the practical picture is that new page loads recover within a minute and existing sessions recover on their next navigation. If the failure is severe enough that existing sessions cannot be left alone, that requires a deliberate reload prompt rather than a shorter TTL — dropping the TTL to zero makes the manifest a per-request fetch on the critical path, permanently, to speed up an event that happens rarely.

// Optional: ask long-lived sessions to reload after a rollback.
setInterval(async () => {
  const current = await fetch('/remotes.json', { cache: 'no-store' }).then((r) => r.json());
  if (current.cart?.rolledBackFrom === loadedVersions.cart) {
    showReloadPrompt('A fix is available. Reload to apply it.');
  }
}, 60_000);

Step 5 — Automate the trigger, keep the decision reviewable #

When automation should act on its own Automatic rollback needs a sustained, attributed signal and a guard against looping when the revert does not fix the problem. Should this trigger an automatic rollback? Sustained breach, version-attributed Roll back automatically 5 minutes over threshold 3× the previous version Single spike Alert, do not act Often a downstream blip A human decides Already rolled back once Do nothing Rollback did not help Escalate instead of looping
The third branch is the one that prevents an automation from flapping between two broken versions all night.

The signal that should trigger a rollback is a sustained, version-attributed regression — not a single spike.

// monitor/auto-rollback.js
const THRESHOLD = 0.02;      // 2% error rate
const SUSTAINED_MS = 5 * 60_000;

async function evaluate(remote) {
  const { version, errorRate } = await currentMetrics(remote);
  const baseline = await previousVersionErrorRate(remote);

  if (errorRate < THRESHOLD || errorRate < baseline * 3) return clear(remote);

  // Only after a sustained breach, and never twice for the same version.
  if (breachDuration(remote) < SUSTAINED_MS) return;
  if (await alreadyRolledBack(remote, version)) return;

  await rollback(remote);
  await notify(`auto-rolled back ${remote} ${version}: ${(errorRate * 100).toFixed(2)}% errors`);
}

Comparing against the previous version’s baseline rather than an absolute number is what prevents a remote with a naturally noisy error rate from rolling itself back every week. The alreadyRolledBack guard prevents a loop where a rollback fails to help and the automation keeps trying.

Step 6 — Handle the cases a rollback does not fix #

Not every bad deploy is fixed by reverting the remote, and reaching for rollback when it will not help costs time.

A remote that persisted bad data — a corrupted entry in localStorage, a malformed value in a shared store — keeps failing on the old version too, because the old version reads the same data. That needs a migration or a clear-and-reload, not a rollback.

A shared dependency version change is not contained by one remote. If the release narrowed a requiredVersion range, other remotes may already have negotiated differently, and reverting one remote can leave the graph in a state neither version was tested against.

A change that spans the seam — a remote and a host that shipped a matched pair — cannot be reverted on one side. This is the strongest practical argument for the additive-change policy in backward-compatible remote API contracts: a remote that only ever adds is a remote that can always be reverted alone.

Step 7 — Rehearse it before you need it #

A rollback path that has never been exercised is a hypothesis.

# .github/workflows/rollback-drill.yml — weekly, against staging
on:
  schedule: [{ cron: '0 9 * * 2' }]
jobs:
  drill:
    steps:
      - run: node scripts/rollback.mjs cart
      - run: npx playwright test e2e/mounted.spec.js   # the page must still work
      - run: node scripts/rollback.mjs cart --to-latest
      - run: npx playwright test e2e/mounted.spec.js

The drill catches the things that quietly break: a pruned asset, a manifest schema change that dropped the history array, an integrity hash that was not carried forward, a permission that expired. Each of those turns a sixty-second recovery into a much longer one, and each is invisible until the day it matters.

Verification #

Troubleshooting #

Symptom: the rollback ran but users still get the broken version.

Diagnosis: the manifest is cached longer than intended, at the CDN or in the browser. Fix: confirm the manifest’s Cache-Control and that no edge rule overrides it; check with curl -sI against the public URL rather than the origin.

Symptom: the previous version 404s.

Diagnosis: a lifecycle rule pruned it. Fix: exclude remote asset paths from lifecycle rules and restore from a build artefact if one still exists.

Symptom: rolling back one remote broke a different one.

Diagnosis: the two negotiated a shared dependency version together. Fix: roll forward with a fix instead, and treat shared-dependency changes as coordinated releases.

Symptom: the next deploy re-shipped the broken build.

Diagnosis: no rollback marker, so the pipeline saw nothing unusual. Fix: record rolledBackFrom and have the publish job refuse that version without an explicit override.

Rollback and roll-forward are different tools #

Teams that get rollback working sometimes reach for it too readily, and the habit has a cost. Reverting is the right response to a regression whose cause is not yet understood, because it restores service while you investigate at a normal pace. It is the wrong response to a small, understood bug with an obvious fix, because a revert discards every other change in that release as well.

The deciding question is whether you know what broke. If you do and the fix is a line, roll forward: publishing a patched version takes about as long as reverting and leaves the release history honest. If you do not, revert first and diagnose from the artefacts — a rolled-back version is still on the CDN, still has its source maps, and can be loaded deliberately in a preview environment using the override described in running remotes locally against a deployed host.

The one case where reverting is unambiguously correct regardless of understanding is a security issue. There, the priority is removing the code from users’ browsers, and every minute spent preparing a careful fix is a minute the affected version is still executing.