How Sentinel works

Sentinel produces findings in two ways: rule checks, which need no language model, and analysis runs, which do. It covers every boundary on the platform, one boundary at a time.

Rule checks

Every two minutes by default, and during every analysis run, Sentinel reads each boundary's pods and records a finding for each of these at once:

Rule	When	Severity
Out of memory	A container was killed for using more memory than its limit	High
Crash loop	A container keeps failing to start	High
Image cannot be pulled	The container's image cannot be fetched or its name is invalid	High
Container cannot be created	The container's configuration stops it from being created	High
No room to run	A pod has waited more than 5 minutes for a node that can take it	Medium
Not accepting traffic	A pod has run for more than 10 minutes without passing its health check	Medium
Restarts repeatedly	A container has restarted 3 times or more	Medium

Only terminations within the last hour count. Copies of the same workload failing the same way make one finding, not one per copy.

Analysis runs

A run starts when Sentinel starts and then every 30 minutes by default. For each boundary it:

1. records the run, which the boundary's Runs tab shows;
2. collects the boundary's evidence: pods that are unhealthy or stopped within the last hour, Kubernetes warning events, Deployments and StatefulSets with fewer ready copies than wanted, the platform's active alerts for the boundary's resources, and connections the platform blocked in the last 24 hours, where it records them;
3. runs the rule checks above;
4. has a language model read the evidence in three stages — Triage reads all of it quickly, Brain reads the most telling part closely, and Code proposes code changes — and records the findings they write;
5. closes the findings it no longer sees — see Findings.

Only one run happens at a time, and the next one starts 30 minutes, by default, after the previous one ended.

The language model is whichever one the platform operator registered and assigned to Sentinel. Without one, runs are skipped: the rule checks still record findings, but no new runs appear in the Runs tab. If the model cannot be reached or its answers cannot be read, the run is recorded as Failed with the reason, and it closes none of the findings the model wrote earlier.

Analyse with logs

For a finding about an app, a static web app or a container instance, a person who may read that resource's logs can choose Analyze with logs. Sentinel reads the recent log lines of the resource's pods — with that person's own permissions — redacts them, and sends them with the finding's evidence to the language model. The model's explanation replaces the finding's summary, impact, steps and verification, and the log lines are kept on the finding as evidence. See Analyse a finding with logs.

Platform findings

Every run also looks at the platform itself: nodes at 85 % or more of their CPU, memory or storage, the platform's own active alerts, and the health of storage volumes. What the model finds there is a platform finding, which belongs to no boundary and is shown only to platform administrators.

Boundaries are kept apart

• Every finding and every piece of evidence belongs to one boundary, or to the platform. It can only be read through that boundary; asking for another boundary's finding answers that it does not exist.
• Each request to the language model carries one boundary's evidence only. A finding the model attributes to evidence that was not in its request stays with the boundary the request was for.

What is removed before anything is stored or sent

Sentinel reads narrow parts of Kubernetes objects: names, labels, container states and resource limits, never environment variables, command lines or annotations. The text it does read — event messages, alert texts and log lines — is redacted as soon as Sentinel collects it, before it is stored or sent to the model. What the model writes back is redacted before it is stored, and every answer Sentinel serves is redacted once more. Redaction removes:

• credentials of known shapes: AWS, GitHub, GitLab, Slack, Stripe and Google keys and tokens, JSON web tokens, private-key headers, bearer tokens, credentials in URLs, and values after password=, secret=, token=, api_key= and similar;
• e-mail addresses, except their domain;
• IPv4 addresses, except 127.0.0.1 and 0.0.0.0;
• host names of three or more parts, except the last two: api.team-a.example.com becomes [REDACTED].example.com.

Everything removed reads [REDACTED].

Caution

Redaction recognises shapes. A secret that looks like ordinary text — a random string with no password= or similar in front of it — passes through, into the finding and to the language model. Keep secrets out of log output.