Observability

Every Mengi Cloud cluster collects metrics, logs and alerts for everything running on it — your deployments, your managed databases and the cluster itself. You can explore the data in the dashboard, query it through the API, and get notified when something needs your attention.

Observability is on by default for every cluster, and each cluster’s data is fully isolated — you only ever see data from your own clusters.

What Mengi collects

Signal	What it covers	Where to view it
Metrics	CPU, memory, pod and node metrics for the cluster and every workload on it	The cluster’s Observability → Metrics tab
Logs	Container logs from all your workloads, plus Kubernetes events	Observability → Logs, and on each deployment’s page
Alerts	Built-in platform alerts plus your own custom alert rules	Observability → Alerts

Where to find it

Per cluster — open a cluster from the Kubernetes page and click Observability. The page has three tabs: Metrics, Logs and Alerts.
Per deployment — every deployment’s page shows its recent logs and recent events from the last hour, refreshed automatically every 30 seconds.
Cluster overview — a cluster’s detail page shows recent events and logs, with a View all link to the full Observability page.

Metrics

The Metrics tab gives you a query box where you write PromQL queries, pick a time range (last 15 minutes up to last 7 days) and run them. Results are shown per series with their labels and values.

If you’re new to PromQL, start from the quick queries:

Quick query	What it shows
Cluster status	Whether all monitored components are up (`up`)
CPU usage	Total CPU used across the cluster
Memory usage	Total memory used across the cluster
Pod count	Number of pods running

Click one to load it into the query box, then adjust it to your needs — for example, scope a query to a single deployment’s namespace (a deployment runs in the namespace app-<deployment-id>).

Querying metrics via the API

The same data is available through the API:

# Instant query — the value right now
curl -G https://api.mengi.cloud/clusters/<cluster-id>/observability/metrics/query \
  -H "Authorization: Bearer $MENGI_API_KEY" \
  --data-urlencode 'query=count(kube_pod_info)'

# Range query — values over a time window
curl -G https://api.mengi.cloud/clusters/<cluster-id>/observability/metrics/query_range \
  -H "Authorization: Bearer $MENGI_API_KEY" \
  --data-urlencode 'query=sum(rate(node_cpu_seconds_total{mode!="idle"}[5m]))' \
  --data-urlencode 'start=2026-06-12T08:00:00Z' \
  --data-urlencode 'end=2026-06-12T09:00:00Z' \
  --data-urlencode 'step=60s'

Timestamps are ISO 8601; step controls the resolution of a range query (default 60s). To discover what you can query, list the available label names and label values:

GET /clusters/<cluster-id>/observability/metrics/labels
GET /clusters/<cluster-id>/observability/metrics/labels/<label>/values

Logs

The Logs tab gives you a query box where you write LogQL queries and a limit (50–1,000 lines in the dashboard). Each result shows the stream’s labels plus timestamped log lines.

A LogQL query starts with a label selector and can add filters:

{namespace="app-<deployment-id>"} — all logs from one deployment
{namespace="app-<deployment-id>"} |= "error" — only lines containing error

Quick queries are available here too, for common namespaces and for filtering on error / warn lines.

Kubernetes events (pod scheduling, restarts, image pulls, …) are collected as a log stream as well — query them with {job="kubernetes-events", namespace="app-<deployment-id>"}. This is the same data the deployment page shows under Recent Events.

Log retention

How long logs are kept depends on the cluster’s tier:

Cluster tier	Retention
Shared	7 days
Dedicated	30 days
Dedicated (production / HA)	90 days

Queries that reach further back than the retention window are automatically clamped to the earliest available data.

Querying logs via the API

curl -G https://api.mengi.cloud/clusters/<cluster-id>/observability/logs/query_range \
  -H "Authorization: Bearer $MENGI_API_KEY" \
  --data-urlencode 'query={namespace="app-<deployment-id>"} |= "error"' \
  --data-urlencode 'start=2026-06-12T08:00:00Z' \
  --data-urlencode 'end=2026-06-12T09:00:00Z' \
  --data-urlencode 'limit=500'

limit can go up to 5,000 entries per request via the API. Instant queries (/logs/query) and label discovery (/logs/labels, /logs/labels/<label>/values) work the same way as for metrics.

Alerts

The Alerts tab shows two things: the active alerts currently firing on the cluster, and the alert rules that produce them.

Each active alert shows its name, a summary of what’s wrong, when it started, and its status. Alerts resolve themselves automatically once the underlying condition clears.

Built-in alerts

Every cluster ships with a set of platform alerts out of the box — you don’t have to configure anything to get them:

Alert	Fires when
Pod crash-looping	A container restarts more than 3 times in 10 minutes
Container out of memory	A container is killed for exceeding its memory limit
Metrics target down	A monitored component stops reporting for 5 minutes
Node memory pressure	A node has been low on memory for 10 minutes
Node disk pressure	A node has been low on disk for 10 minutes
Volume nearly full	A persistent volume drops below 15% free space
Certificate expiring soon	A TLS certificate expires in under 14 days and hasn’t renewed

Built-in alerts fire at warning severity and appear in the Alerts tab like any other alert.

Custom alert rules

You can add your own rules from the Alerts tab (Create Rule) or via the API. A rule consists of:

Name — e.g. High CPU Usage.
Expression — a PromQL condition that should fire, e.g. sum(rate(node_cpu_seconds_total{mode!='idle'}[5m])) > 0.8. Anything you can query in the Metrics tab can be alerted on.
Duration — how long the condition must hold before the alert fires (a number plus a unit: 30s, 5m, 1h). This avoids alerting on short blips.
Severity — critical, warning or info.

Via the API you can additionally attach labels and annotations (e.g. a summary shown with the alert), enable/disable a rule without deleting it, and link it to notification channels:

curl -X POST https://api.mengi.cloud/clusters/<cluster-id>/observability/alerts/rules \
  -H "Authorization: Bearer $MENGI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "High memory usage",
    "expression": "sum(container_memory_working_set_bytes{namespace=\"app-<deployment-id>\"}) > 4e9",
    "duration": "10m",
    "severity": "warning",
    "annotations": {"summary": "App is using more than 4 GB of memory"},
    "notification_channel_ids": ["<channel-id>"]
  }'

Rules are updated with PUT .../alerts/rules/<rule-id> and deleted with DELETE .../alerts/rules/<rule-id> (or the delete button in the dashboard).

Silencing alerts

When you’re working on a known issue and don’t want to keep being notified, silence the alert instead of deleting its rule:

On an active alert, click Silence.
Pick a duration (15 minutes up to 1 week in the dashboard).
Add a comment explaining why — it’s required, so your future self (and teammates) know what happened.

A silenced alert stops notifying but keeps being evaluated; when the silence expires, it fires again if the condition still holds. Via the API you can list a cluster’s silences (GET .../alerts/silences) and end one early (DELETE .../alerts/silences/<silence-id>).

Notification channels

Notification channels define where alerts are sent. They belong to your account, so you create them once and reuse them across clusters and rules. Channels are currently managed via the API.

Type	Configuration
email	`email_addresses` — a list of recipient addresses
slack	`webhook_url` — a Slack incoming-webhook URL
webhook	`url` — any HTTP(S) endpoint, plus optional `headers`
pagerduty	`routing_key` — a PagerDuty Events integration key

Creating a channel

curl -X POST https://api.mengi.cloud/notification-channels \
  -H "Authorization: Bearer $MENGI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "On-call Slack",
    "type": "slack",
    "config": {"webhook_url": "https://hooks.slack.com/services/..."}
  }'

Testing a channel

Before relying on a channel, send a test notification and check it arrives:

curl -X POST https://api.mengi.cloud/notification-channels/<channel-id>/test \
  -H "Authorization: Bearer $MENGI_API_KEY"

Routing alerts to channels

Attach channels to an alert rule by setting its notification_channel_ids when creating or updating the rule. When the rule fires, every linked, enabled channel is notified. Disable a channel ("is_enabled": false) to pause its notifications without unlinking it from your rules.

Good to know

Data is isolated per cluster — metrics, logs and alerts never cross cluster (or account) boundaries.
A deployment’s namespace is app-<deployment-id> — use it to scope metric and log queries to one app. Imported deployments keep their original namespace instead.
Log retention follows the cluster tier (7 / 30 / 90 days) — pick a higher tier if you need a longer history.
Silence, don’t delete — silencing keeps the rule and its history; deleting a rule loses the configuration.
Observability can be turned off per cluster via the API (PATCH /clusters/<cluster-id>/observability/config with {"observability_enabled": false}) — the dashboard shows a banner when collection is disabled.
Lists returned by the API include a total count; alert rules, silences and channels are all plain JSON you can manage from scripts or CI.