3.6 Operational Troubleshooting
3.6.1 Service Dependency Map
The dependency direction below makes it possible to localise a fault rather than restarting everything. From 3.1.1: CBOM-Server depends on the database and identity provider; Collector-Services depends on the database; discovery and deployment depend on CBOM-Server for task definitions and on the collector for delivery; the frontend depends on CBOM-Server; the observability pipeline depends on nothing, and everything depends on it for visibility.
3.6.2 Common Failure Scenarios
Portal loads but all data views are empty
Check the database connection from CBOM-Server, then whether the seeding step ran, then whether any discovery has actually completed. An empty inventory on a new install is expected, not a fault.
Users are logged out unpredictably
Check identity provider reachability, then clock skew between the platform and the provider, then whether session renewal is succeeding. A network error should not end a session — it produces a connection error, not an expiry — so an expiry dialogue during a network blip indicates a client-side handling problem rather than a genuine session end.
Scans run but nothing appears in the inventory
Confirm submissions are reaching the collector, then that they authenticate, then that they pass schema validation, then that the deduplication pass is running. Check the ingestion accounting: a high skipped count localises the fault to validation, and a zero requested count localises it to the sensor never having submitted at all.
A material appears with the wrong host or path
Check the source properties on the underlying documents. Provenance is recorded by the sensor at discovery time, so an incorrect host or path is a sensor configuration problem, not a consolidation problem.
Duplicate materials in the inventory
Duplicates mean two documents produced different canonical identities for what should be the same material — usually a differing content hash because one sensor read a different representation of the same object. Check whether both sensors are at the expected version.
A dashboard widget shows no data while others work
Check the widget parameters, then whether the underlying material type exists in the inventory, then the widget definition itself. Widget definitions live in the database, so a widget can be misconfigured without any code change.
An integration stops working
Check the integration credential first — an expired client secret or rotated API key is the usual cause — then network reachability to the endpoint, then whether the far end has changed its API.
3.6.3 Discovery-Specific Failures
A task fails against one target and succeeds against others — credential or reachability problem specific to that target.
-
A task fails against all targets: template configuration or a platform-side problem.
A sensor returns no findings where findings are expected — check the sensor scope configuration first, then whether the account has enough privilege to see what is being looked for. A sensor with insufficient privilege reports success and no findings, which is indistinguishable from a clean result.
A remote deployment succeeds but no results arrive — the sensor ran and could not reach the collector. Check the emitter configuration and the network path, and look for buffered result files on the target.
-
A scan takes far longer than expected: check the scope. A filesystem or source scan with an over-broad scope is the usual cause.
3.6.4 Performance Troubleshooting
-
Resource utilisation first: CPU, memory, disk and disk I/O on each tier. Discovery is CPU-bound; the database is I/O-bound.
-
Slow dashboards: usually database query performance. Check for missing indexes, particularly after an upgrade, and check inventory size against the widget query.
-
Slow or backlogged ingestion: check queue depth if queue-based, collector resource usage, and database write performance.
-
Long deduplication passes: scale with the number of staged documents per cycle. Incremental scanning is the remedy; more frequent smaller scans is the other.
-
Slow scans: scope first, then network latency to targets, then discovery tier resources.
Platform load is driven by scan frequency and scope, not by inventory size. Scaling up a scan schedule means scaling the discovery and ingestion tiers, not the data tier.
3.6.5 Support Bundle Collection
Define what to collect before support asks for it, and define what must be removed before it is sent.
Platform version, deployment topology and deployment method.
Service status for every component.
-
Logs from the relevant window: central store export, plus per-service on-disk logs if the pipeline itself is implicated.
Configuration with all secrets redacted.
The ingestion accounting rows covering the affected period.
-
For a discovery problem: the task and template definition, the target platform and version, and the sensor versions involved.
-
For a data problem: the canonical identity of the affected material and its history entries.
Before sending the bundle, remove credentials, tokens, keys and the credential encryption key.
3.6.6 Where to Look First
The sections above set out what goes wrong and why. These add the commands and the reads that answer it.
Four reads cover most questions, in the order worth trying them.
| Read | Answers |
|---|---|
| Service status | Is everything running |
| The System Logs page | What the platform is saying about itself |
| The deployment history for the task in question | Did the scan deliver, and how much |
| The audit trail | Who changed what, and when it stopped working |
The last is the one people reach for too late. A platform that worked on Friday and not on Monday has usually been changed, and the audit trail says by whom.
3.6.7 Checking Service Status
# Container deployment
docker compose ps
docker compose logs --tail=100 cbom-server
docker compose logs -f collector-worker
# Native deployment
./start.sh cbom | collector | collector-worker | ai | frontend
./stop.sh cbom | collector | collector-worker | ai | frontendA container in the running state is not a service that is serving. Confirm with the version endpoint, which is also what the platform's own health check calls:
curl -fsS https://cbom.example.com/cbomsrv/api/v1/versionThe discovery service and the AI service each expose their own health path; the database and the broker answer their native ping commands.
3.6.8 Reading the Logs
The platform writes one log per domain rather than one log for everything, so a question about discovery does not require reading authentication noise.
| Log | Carries |
|---|---|
auth | Sign-in, session creation and renewal |
admin | User, role and permission changes |
host | Host records and credentials |
discovery | Task execution and sensor dispatch |
ingest | Agent submissions and their validation |
cmdoc | Material writes and deduplication |
policy | Policy evaluation |
config | Configuration load, save and distribution |
auth_framework | The identity framework underneath sign-in |
Paths are relative to the installation root, so one configuration is correct both in a container and in a checkout. Each record is a single JSON object, so the fields are real keys a log tool can filter on rather than text that merely contains them.
An empty log directory does not mean a quiet system. If a handler cannot open its file the logging subsystem degrades to a fallback — and the directory is created anyway, empty. So "no log files at all" is a configuration symptom, not silence. Check the platform's own logging health, which reports which framework loaded and where the handlers actually landed.
The System Logs page reads the same records with filtering and faceting, and is the better starting point unless you are on the host anyway.
3.6.9 The Ingestion Counters
Almost every "the scan ran and nothing appeared" question is answered by four numbers, returned by the deployment call and kept in the deployment history.
| Counter | Zero means |
|---|---|
requested_count | The sensor never submitted anything — the problem is on the target side |
inserted_count | Nothing new; with a non-zero updated count this is normal on a re-scan |
updated_count | Nothing matched an existing material |
skipped_count | Non-zero means documents were rejected and discarded |
skipped_count is the one to act on. A document that fails validation leaves no other trace — this counter is its only record. A non-zero value localises the fault to validation, and the ingest log names the field.
3.6.10 Recovering a Node
An application node. Stateless apart from live sessions. Restart it; users signed in against that instance will sign in again. On a multi-node tier, drain it at the load balancer first.
A collector worker. Safe to restart at any time. Unacknowledged messages return to the broker and are redelivered, and ingestion is idempotent on resubmission, so a worker killed mid-batch costs a redelivery rather than data.
A broker node. Agents spool to their own disk while the broker is unreachable and republish when it returns, so an outage delays a scan rather than destroying one. On a cluster, restart one node at a time and wait for the quorum queues to report synchronised.
A database member. The procedure is in High Availability Runbooks. Establish first whether the set still has a majority: with one, the platform is running and you have time; without one it is read-only and you do not.
The identity provider. Existing sessions continue until they expire; new sign-ins fail while it is down. This is why the outage reads as intermittent — people already signed in notice nothing.
3.6.11 Stalled Scans
A scan can stall without failing: a chunk that was dead-lettered, or an agent whose broker connection died mid-scan, leaves the scan permanently incomplete.
The platform sweeps for this on its own. A scan quiet for longer than the staleness window is picked up and completed, and the sweep also notices deposits that arrived by a route with no progress record at all. In practice a stalled scan resolves itself within the sweep interval and needs no operator action.
What it does not do is recover documents the broker dropped. If a scan's results are genuinely gone, re-run the task.
3.6.12 Diagnosing by Elimination
The dependency direction in 3.6.1 is what makes this tractable. Working from the bottom up:
Is the database reachable from the application host, with the application's own credentials? Not from the database host as root — that tests something else and usually succeeds.
Is the identity provider answering? If not, nobody new can sign in and everything else looks fine.
Is the backend serving? The version endpoint.
Is the broker accepting and the worker consuming? The broker's management portal shows queue depth; a depth that only grows means nothing is consuming.
Is the proxy routing? A 502 on one path and a 200 on another localises it immediately.
Only after those five does it make sense to look at a sensor, a template or a widget.