Configuration

Configuration

This will be a lot of copy/paste but don't be lazy and try to understand what we are doing here.

Prometheus

Let’s edit the Prometheus config file and add all the modules to it:
1
sudo nano /etc/prometheus/prometheus.yml
Copied!
Add the following code to the file and save ctrl+o ctrl+x:
1
global:
2
scrape_interval: 15s
3
evaluation_interval: 15s
4
5
rule_files:
6
- 'rules.yml'
7
8
alerting:
9
alertmanagers:
10
- static_configs:
11
- targets:
12
- localhost:9093
13
14
scrape_configs:
15
- job_name: "prometheus"
16
scrape_interval: 5s
17
static_configs:
18
- targets: ["localhost:9090"]
19
- job_name: "substrate_node"
20
scrape_interval: 5s
21
static_configs:
22
- targets: ["localhost:9615"]
23
- job_name: "node_exporter"
24
scrape_interval: 5s
25
static_configs:
26
- targets: ["localhost:9100"]
27
- job_name: "process-exporter"
28
scrape_interval: 5s
29
static_configs:
30
- targets: ["localhost:9256"]
Copied!
  • scrape_interval defines how often Prometheus scrapes targets, while evaluation_interval controls how often the software will evaluate the rules.
  • rule_files set the location of Alert manager rules we will add next.
  • alerting contains the alert manager target.
  • scrape_configs contain the services Prometheus will monitor.

Alert rules

Let’s create the rules.yml file this will set the rules for Alert Manager:
1
sudo touch /etc/prometheus/rules.yml
2
sudo nano /etc/prometheus/rules.yml
Copied!
We are going to create 2 basic rules that will trigger an alert in case the instance is down or the CPU usage crosses 80%. Add the following lines and save the file:
1
groups:
2
- name: alert_rules
3
rules:
4
- alert: InstanceDown
5
expr: up == 0
6
for: 5m
7
labels:
8
severity: critical
9
annotations:
10
summary: "Instance $labels.instance down"
11
description: "[{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 1 minute."
12
13
- alert: HostHighCpuLoad
14
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80
15
for: 0m
16
labels:
17
severity: warning
18
annotations:
19
summary: Host high CPU load (instance Astar Node)
20
description: "CPU load is > 80%\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
Copied!
The criteria for triggering an alert are set in the expr: part. To create your own alerts, you’re going to have to learn and test the different variables provided to Prometheus by the services we are setting up. There is (almost) an infinite number of possibilities to personalize your alerts.
As this part can be time-consuming to learn and build, you can find a summary list of alerts we like to use. Feel free to share your Alert file with the community. You should also have a look at alerts provided by Parity.
Then, check the rules file:
1
promtool check rules /etc/prometheus/rules.yml
Copied!
And finally, check the Prometheus config file:
1
promtool check config /etc/prometheus/prometheus.yml
Copied!

Process exporter

Process exporter needs a little config file to be told which processes they should take into account:
1
sudo touch /etc/process-exporter/config.yml
2
sudo nano /etc/process-exporter/config.yml
Copied!
Add the following code to the file and save:
1
process_names:
2
- name: "{{.Comm}}"
3
cmdline:
4
- '.+'
Copied!

Gmail setup

To allow AlertManager to send an email to you, you will need to generate something called an app password in your Gmail account. For details, click here to follow the whole setup.
You should see something like the below:
Copy and save it somewhere

Alert Manager

There is a configuration file named alertmanager.yml inside the directory that you just extracted in the previous command, but that is not of our use. We will create our alertmanager.yml file under /etc/alertmanager with the following config.
Let’s create the file:
1
sudo touch /etc/alertmanager/alertmanager.yml
2
sudo nano /etc/alertmanager/alertmanager.yml
Copied!
And add the Gmail configuration to it and save the file:
1
global:
2
resolve_timeout: 1m
3
4
route:
5
receiver: 'gmail-notifications'
6
7
receivers:
8
- name: 'gmail-notifications'
9
email_configs:
10
- to: YOUR_EMAIL
11
from: YOUR_EMAIL
12
smarthost: smtp.gmail.com:587
13
auth_username: YOUR_EMAIL
14
auth_identity: YOUR_EMAIL
15
auth_password: YOUR_APP_PASSWORD
16
send_resolved: true
Copied!
With the above configuration, alerts will be sent using the email you set above. Remember to change YOUR_EMAIL to your email and paste the app password you just saved earlier to the YOUR_APP_PASSWORD.
We will test the Alert Manager later in the guide.