Prometheus Alertmanager alerts on your iPhone

Each firing Alertmanager group arrives as one notification from one webhook receiver, and the same notification changes when it clears.

What you need#

• Nowlark on your iPhone, and your key from Settings › Connect an App › Create Your Key. Copy it when it is shown, because it is shown once. It starts nlk_.
• Prometheus Alertmanager, and access to its configuration file.

Add Nowlark to Alertmanager#

1. Open alertmanager.yml on the machine that runs Alertmanager.
2. Add this receiver to it, with your own key in place of nlk_…:

receivers:
  - name: nowlark
    webhook_configs:
      - url: https://nowlark.com/v1/hooks/alertmanager
        send_resolved: true
        http_config:
          authorization:
            credentials: nlk_…

3. Route the alerts you want on your iPhone to it, for example receiver: nowlark on your top-level route.
4. Reload Alertmanager, with kill -HUP or a POST to /-/reload.
5. To check it, fire a test alert, with your own Alertmanager's address in place of localhost. It arrives on your iPhone at High priority within your route's group_wait.

amtool alert add alertname=NowlarkTest severity=warning \
  --alertmanager.url=http://localhost:9093

Your key goes in the Authorization header, not in the address. To keep it out of the file, use credentials_file instead of credentials.

What arrives#

The alert's severity label sets the priority: critical or page is Urgent and breaks through Focus modes, warning, error or high is High, and resolved is Normal. An alert with no severity label arrives at High.

Each alert arrives under the source Alertmanager. In the app's Sources tab you can give it a colour, and with the Upgrade Pass its own tone.

On a team channel#

Use a channel key instead of your own, and everyone on the channel gets the alert. When Alertmanager reports it resolved, the alert is resolved on the channel too, and no later escalation step goes out for it. Channels and escalation covers the rest.

Get the app