High Availability Runbooks
The procedures for building each resilience tier, and for the operations you will perform on one afterwards. High Availability and Scaling describes the shapes and what fails independently of what; this page is the steps.
Every runbook here assumes a working single node — see Quick Start — and takes it as the starting point.
Before anything else: the session constraint
Read this before designing a load-balanced tier, because it decides how the load balancer must be configured.
Sessions are held in memory by the backend process that created them. They are not in the database and not in a shared cache. A session identifier minted by one backend instance is unknown to every other instance.
Two consequences, both of which look like intermittent faults if you do not expect them:
A load balancer in front of two or more backend instances must use session affinity. Round-robin across instances produces a 401 on roughly every request that lands on an instance other than the one that signed the user in. The console recovers by refreshing, which hides the problem until it does not.
Restarting a backend instance signs out everyone it was serving. Plan rolling restarts for a quiet window, or accept that users will sign in again.
Affinity on the session cookie or the Authorization header value is sufficient. This constraint applies only to the backend API; the collector, the console and the AI service are genuinely stateless and can be balanced any way you like.
Runbook 1 — Separate the database (Tier 1 to Tier 2)
The highest-value single step, and the one to take first. It stops an analysis pass and a database compaction competing for the same memory.
1. Provision the database node with the memory and disk from the Sizing guide. The working set is what matters, not the total size.
2. Install MongoDB and bind it to the platform-facing interface only. Do not bind 0.0.0.0 — that exposes the whole database with only the root password in front of it.
# /etc/mongod.conf
net:
port: 27017
bindIp: 127.0.0.1,10.20.30.50
security:
authorization: enabled3. Create the application user with rights on the two databases and nothing else:
use admin
db.createUser({
user: "cbomapp",
pwd: "<strong password>",
roles: [
{ role: "readWrite", db: "CBOMData" },
{ role: "readWrite", db: "CBOMAudit" }
]
})4. Open the firewall from the application node to port 27017, and from nowhere else.
5. Move the data. Stop the application services, dump, restore, verify:
docker compose stop cbom-server collector collector-worker discovery-server ai-service
docker compose exec -T mongodb mongodump --archive --gzip \
--username "$MONGO_ROOT_USER" --password "$MONGO_ROOT_PASSWORD" \
--authenticationDatabase admin > cbom-migrate.archive.gz
mongorestore --archive --gzip \
--uri "mongodb://cbomapp:<password>@10.20.30.50:27017/?authSource=admin" \
< cbom-migrate.archive.gzCompare document counts in both CBOMData and CBOMAudit before going further. CBOMAudit is the one people forget — it holds the audit trail, the material histories and the deployment histories, and it is a separate database.
6. Repoint the platform. Update the database URI in .env, remove the mongodb service from the compose file, and start the application services again.
7. Verify that the inventory, the dashboards, the audit trail and the material history all read correctly. The history pages are the ones that prove CBOMAudit came across.
Runbook 2 — Convert the database to a replica set
A three-member set tolerates the loss of one member and elects a new primary without operator action.
1. Provision two more database nodes, identically sized. Put them in three failure domains — three availability zones if you have them, three racks if you do not. Two zones does not help: losing the zone with two members leaves one, which is not a majority.
2. Create a keyfile and copy it, with identical content and ownership, to all three:
openssl rand -base64 756 > /etc/mongo-keyfile
chmod 400 /etc/mongo-keyfile && chown mongodb:mongodb /etc/mongo-keyfile3. Configure all three with the same replica set name and the keyfile, then restart each:
replication:
replSetName: cbom-rs
security:
authorization: enabled
keyFile: /etc/mongo-keyfile4. Initiate the set from the node holding the data:
rs.initiate({
_id: "cbom-rs",
members: [
{ _id: 0, host: "db01.corp.example.com:27017", priority: 2 },
{ _id: 1, host: "db02.corp.example.com:27017", priority: 1 },
{ _id: 2, host: "db03.corp.example.com:27017", priority: 1 }
]
})5. Wait for the initial sync and confirm with rs.status() that one member is PRIMARY and two are SECONDARY.
6. Repoint the platform at the set, not at a member. This is the step that makes the failover automatic:
mongodb://cbomapp:<password>@db01.corp.example.com:27017,db02.corp.example.com:27017,db03.corp.example.com:27017/CBOMData?replicaSet=cbom-rs&authSource=adminA URI naming one member still works and still fails over nothing — the driver has no way to find the new primary.
7. Test the failover before you need it. rs.stepDown() on the primary, then confirm the console keeps working. A failover that has never been exercised is an assumption.
Runbook 3 — Add a second application node
1. Provision the node and install Docker with the Compose plugin.
2. Copy the compose file and .env, then remove the services that must not be duplicated — the database, the identity provider and the message broker each have their own clustering procedure, and running a second independent copy of any of them splits your data.
What to replicate per application node: the backend API, the collector, the collector worker, the console, the AI service and the reverse proxy.
3. Point both nodes at the shared database, broker and identity provider. Every address in .env that currently says localhost or a container name has to become a resolvable address of the shared service.
4. Put a load balancer in front, with:
Health checks on each backend instance, so a failed instance is removed rather than serving errors.
Session affinity on the backend API — see the constraint at the top of this page.
TLS terminated either at the load balancer or passed through; if terminated there, the certificate must carry the public name and the backend must be told it is behind a proxy.
5. Update the public addresses in .env on both nodes to the load balancer's name, and rebuild the console so its compiled backend URL matches.
6. Verify by signing in, then stopping one node's backend. The console should keep working. Users who were affine to the stopped instance will be signed out — that is the expected behaviour, not a fault.
Runbook 4 — Cluster the message broker
The queue the agents publish to is a quorum queue, which replicates across the cluster and survives the loss of a minority of nodes.
1. Provision three broker nodes. Three, not two: a quorum queue needs a majority.
2. Give all three the same Erlang cookie, with identical content and 400 permissions.
3. Join them into a cluster, resetting each joining node first:
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@broker01
rabbitmqctl start_app4. Confirm with rabbitmqctl cluster_status that all three are running.
5. Point the collector workers at all three, so a worker reconnects to a surviving node rather than waiting for its own to come back.
6. Point the agents at a stable address — a load balancer or a DNS name covering all three. The broker address travels in each agent's configuration, so changing it later means redeploying every agent. Decide this before the first large deployment.
Agents spool to their own disk when the broker is unreachable , and republish when it returns. A broker outage delays a scan; it does not destroy one. This is what makes broker maintenance an ordinary operation rather than a scheduled outage.
Runbook 5 — Rolling restart without an outage
On a multi-node application tier:
Drain one node at the load balancer and wait for its connections to finish.
Restart its services:
docker compose up -d --force-recreate.Wait for its health check to pass.
Return it to the pool.
Repeat for the next node, one at a time.
Sessions held by a restarted backend are lost — see the constraint at the top. Everything else continues.
For the database, restart secondaries first, then step the primary down and restart it last. For the broker, restart one node at a time and wait for the quorum queues to report synchronised before taking the next.
Runbook 6 — Recover a failed database member
Confirm the set still has a majority:
rs.status(). With two of three healthy, the platform is running and you have time.Provision a replacement with the same configuration, replica set name and keyfile.
Remove the dead member:
rs.remove("db02.corp.example.com:27017").Add the replacement:
rs.add("db02new.corp.example.com:27017").Watch the initial sync complete, then confirm three healthy members.
If the set has lost its majority it is read-only and the platform is effectively down. Restore a member from backup and reconfigure the set to the surviving members with rs.reconfig(..., {force: true}). This is the scenario to have rehearsed — see Backup and Recovery.
Verifying a tier
Whatever tier you have built, prove it with the failure it is supposed to survive:
| Tier | Test | Expected |
|---|---|---|
| 2 | Stop the application node | Platform down; database intact; restarting restores service with no data loss |
| 3 | Stop one application node | Console keeps working; affine users sign in again |
| 3 | rs.stepDown() on the primary | Brief pause, then normal service |
| 3 | Stop one broker node | Publishing continues |
| 4 | Fail a whole zone | Service continues in the others |
A tier that has not been tested is a diagram. Schedule the test, write down what actually happened, and fix the gap between that and this page.