Observability
The operator receives a dashboard, a set of alerts with a configurable destination, and access to the logs. This page describes what each one measures and what to do when an alert fires.
Dashboards
- GPU: utilization, memory, temperature, and power draw per device; usage per lab and per model in service. It is the same information that the Client administrator sees summarized in the infrastructure dashboard.
- Queues: jobs waiting, admitted, and preempted per queue; average wait time.
- Platform: status of each component, errors per service, API latency.
- InferNode: requests, tokens, and credits per Client and per model; errors per provider.
Alerts
Alerts are grouped by severity and go to one channel per severity (by default two chat channels, one for warning and another for critical; they can be routed to email, Teams, or another webhook). Each one has a confirmation time (for) so that it does not fire on spikes that last a few seconds.
| Alert | Condition | For | Severity | What to check |
|---|---|---|---|---|
| Component restarting | A platform service restarts in a loop. | 2 min | Critical | Component logs; almost always a configuration or a secret that stopped being valid. |
| Component with no ready replicas | A platform service is not ready. | 5 min | Warning | Cluster status; if it persists, escalate to Saptiva. |
| Replicas mismatch / deployment stuck | A deployment does not reach its replicas or does not progress. | 10 / 15 min | Warning | It usually accompanies a half-finished upgrade; see Upgrade. |
| Quota almost full / full | A lab consumes more than the threshold of its budget or exhausts it. | 15 min | Informational | Nothing to fix in the platform: it is a user's lab. Notify the Client administrator if it repeats (Quotas). |
| High ingress latency | p95 of requests above 2 s on a route. | 15 min | Warning | GPU load or saturated model; correlate with the InferNode dashboard. |
| High authentication latency | Session verification takes longer than normal. | 5 min | Warning | Slow identity provider or loaded identity database. |
| High error rate | More than a threshold of 5xx responses on platform routes (a separate rule for the lab route). | 5 / 15 min | Critical | Logs of the service returning 5xx; if it is InferNode, the model provider. |
| Certificate expiring | A certificate shows less than 21 days of validity. | 30 min | Warning | Renew it. The Client's certificates do not renew on their own: they are loaded into the secrets store. |
| Critical certificate | Less than 7 days. | 30 min | Critical | Renew it now to avoid an outage visible to users. |
| Volume almost full | A volume exceeds 85% usage. | 5 min | Warning | If it belongs to a lab, notify the owner; if it belongs to the model catalog or to artifacts, expand or clean up. |
| Secret not synced | A secret in the store could not be read for 15 min. | 15 min | Critical | Connectivity or credential of the secrets store; until it is resolved, rotations are not applied. |
| Metrics shipping failed | Metrics are not being archived. | 10 min | Critical | Cluster object storage. |
| Autoscaling inactive or impossible | A service cannot scale. | 15 min | Warning | Lack of CPU or memory capacity on the nodes. |
| Pod evictions | The node evicted processes due to lack of resources. | Immediate | Warning | Memory or disk pressure on a node; check which lab caused it. |
| Memory page faults | A node pages to disk at a high rate. | 60 min | Warning | Overloaded node; move load or expand memory. |
| Backup failed | The daily backup did not finish. | Critical | Backup procedure (Upgrade and backup). |
Two alerts that are not in the base set and that are worth requesting in installations with many labs: jobs waiting for a GPU longer than a set time, and a GPU that stops reporting metrics. Both are derived from the GPU dashboard data.
How to report a problem to Saptiva AI
A report with this information gets resolved without back-and-forth:
- Which alert or symptom, with the time and time zone.
- Client and, if applicable, the affected lab or workflow.
- The request identifier (InferNode
X-Request-Id) or run identifier (visible in the workflow trace). - A screenshot of the dashboard or of the message in Studio.
Logs
Platform logs are kept in the cluster for a configurable period (14 days by default) and can be sent to the Client's logging system. Each request to InferNode and each workflow run carry an identifier that appears in the logs and in the responses to the user, to correlate reports.