Monitoring and Alerts
Volāre uses several monitoring tools, each for a different question:
| Question | Tool | Where |
|---|---|---|
| Is a server, container, the database or the error rate unhealthy? | Grafana Cloud (metrics, logs, alerts) | volare.grafana.net |
| Why is a request, job or query slow in Laravel? | Laravel Nightwatch | nightwatch.laravel.com |
| What exception happened, with stack trace? | Sentry | sentry.io |
| Is the app’s own health check passing? | Spatie Health | /health-dashboard (Health Monitoring) |
| What did a request do locally? | Telescope | http://localhost/telescope |
How data reaches Grafana
Section titled “How data reaches Grafana”Every server image ships Grafana Alloy, which collects:
- server metrics (CPU, memory, disk, load) and container metrics (cAdvisor, built into Alloy);
- container logs, including nginx access logs, plus the system journal.
Each series is labelled with the environment, the server role (api, front, ops-runner) and
the host name (<asg>-<az>-<instance-id>). RDS and load-balancer metrics come from CloudWatch,
which Grafana reads through a cross-account IAM role.
Dashboards are JSON files in infra-volare/grafana/dashboards/ (infrastructure, containers, logs,
availability, costs). They are imported by hand, not managed by Terraform.
Alerts
Section titled “Alerts”Alert rules are code in infra-volare/terragrunt/modules/grafana-alerting/ and are applied
automatically when merged to main. Only production is alerted, and every alert is critical.
| Group | Fires when |
|---|---|
| Node | Server CPU, memory or disk above 95 %, load above 5×, or a server rebooted |
| Container | A container uses over 95 % CPU or too much memory |
| Container restarts | A container restarted more than 3 times in 15 minutes |
| Expected service | An expected container (php-fpm, web, queue-worker, scheduler, nightwatch, analytics-mcp, front) has not been seen for 5 minutes |
| HTTP | 5xx responses above 20 % (with at least 20 requests) or above 20 in 5 minutes |
| Service capacity | An ASG has fewer healthy instances or load-balancer targets than desired |
| RDS | Database CPU above 80 %, more than 650 connections, under 5 GiB free storage or under 1 GiB free memory |
| Log pipeline | Alloy stopped shipping logs |
| Hermes | Hermes was blocked from delivering a file |
Alerts go to email, Telegram and Google Chat (contact point all-alerts). Critical alerts repeat
every hour until resolved.
When an alert fires
Section titled “When an alert fires”The runbook is infra-volare/docs/runbooks/grafana-alerts.md. The first steps are always the same:
- Check for related alerts and for a deploy in the last hour (infra-volare Actions).
- Look at the matching Grafana dashboard and the container logs (Loki) before changing anything.
- Prefer reversible fixes: a rollback deploy of the previous tag, or an instance refresh.
- Do not raise ASG capacity to hide unhealthy targets — find out why they are unhealthy.
For database alerts use RDS Performance Insights and pg_stat_activity. For application errors
start in Sentry and Nightwatch. To get onto a server, see Server Access.