3.2 Logging and Log-Based Monitoring
3.2.1 Logging Architecture
Services use standard structured logging, routed through an OpenTelemetry collector into Loki, and read through Grafana. The practical consequences for an administrator are: logs are queryable across services from one place, log retention is configured centrally rather than per service, and logs survive the container that produced them.
3.2.2 Log Locations
-
Central: the log store, queried through the log dashboard. This is the primary place to look.
-
Per service on disk: each backend writes rotating logs to its own log directory, held in a persistent volume in a container deployment. Retention on these is time-based, and the window is configurable.
-
Supporting services: database, identity provider, queue and proxy logs are read through their own standard mechanisms.
When the observability pipeline itself is the problem, the per-service on-disk logs are the fallback. This is the one case where the central log store cannot be trusted.
3.2.3 Log Retention
Central store retention is configured in the log store configuration file, mounted from the host and editable without an image rebuild.
Per-service on-disk logs rotate on a time basis with a documented retention window.
Retention should be set to at least the customer incident investigation window, and separately to whatever their audit obligations require. These are usually different numbers, and the audit obligation is usually the longer one.
3.2.4 Recommended Log-Based Alerts
The conditions below are the ones worth alerting on, grouped by component, each with the reason it matters.
Database
-
Connection failures from either backend: ingestion and the portal are both about to fail.
Replica set member unavailable or election events — availability is degraded even if service appears normal.
-
Storage approaching capacity: the database is the first thing to fail on a full disk, and it fails badly.
-
Sustained slow queries: usually a missing index after an upgrade.
Identity provider
-
Authentication failures spiking: either an attack or a broken configuration; both need attention.
-
Token validation failures from the backend: a clock skew or certificate problem, and it will present as random logouts.
-
Provider unreachable from the backend: new logins are failing while existing sessions continue, so users report the problem inconsistently.
Ingestion
-
Authentication failures at the collector: an expired or revoked host token, or an unregistered host attempting to submit.
-
Schema validation rejections: a sensor version mismatch. Silent data loss otherwise, because the sensor believes it submitted successfully.
Non-zero skipped counts in the ingestion accounting — documents arrived and were not stored.
Ingestion volume dropping to zero for a host that normally reports — a scan that has silently stopped running is invisible without this alert. This is the single most valuable alert in the set.
Message queue
-
Queue depth growing without draining: the collector is not consuming.
Connection failures from emitters or the collector.
Disk alarms on the queue node.
Application
Unhandled exceptions in either backend.
-
Deduplication pass failures: the inventory is not being updated even though ingestion succeeds, so the portal shows stale data with no error.
-
Configuration write failures: a configuration change that reported success but did not persist.
-
Integration authentication failures: a rotated credential or expired client secret at the far end.
Scan task failures, by task — a task failing repeatedly against the same target is a coverage gap.
Observability pipeline
-
Collector buffer growth or drops: logs are being lost.
Log store ingestion errors or rejections.
No logs received from a service that should be emitting — the service is down, or its logging is broken.