Prometheus - Installation and Configuration

Author	Date
Emilia	21.02.2025

Updated date	Updated by
1.03.2025	Emilia

Monitor Main Server Guidelines

Setup the 'main' monitoring server. For clients, see chapter below.

# Create the Prometheus & Node-exporter Users
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

sudo groupadd --system node_exporter
sudo useradd -s /sbin/nologin --system -g node_exporter node_exporter

# verify users
getent passwd prometheus
getent passwd node_exporter

# install 
sudo apt update

cd /tmp

wget https://github.com/prometheus/prometheus/releases/download/v3.1.0/prometheus-3.1.0.linux-amd64.tar.gz
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

tar -xvf prometheus-3.1.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz

sudo mv prometheus-3.1.0.linux-amd64/prometheus /usr/bin/
sudo mv prometheus-3.1.0.linux-amd64/promtool /usr/bin/
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo mkdir /usr/share/prometheus

sudo chown prometheus:prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/share/prometheus
sudo chown prometheus:prometheus /usr/bin/prometheus
sudo chown prometheus:prometheus /usr/bin/promtool

# Verify Services [possibly too early]
# sudo systemctl is-enabled prometheus
# sudo systemctl status prometheus
# sudo systemctl is-enabled prometheus-node-exporter
# sudo systemctl status prometheus-node-exporter

Configure

Run: sudo vim /etc/prometheus/prometheus.yml and insert:

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: '$HOSTNAME'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['127.0.0.1:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'node'
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['127.0.0.1:9100']

Run: sudo vim /lib/systemd/system/prometheus.service and insert:

[Unit]
Description=Monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
Requires=network-online.target
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
WorkingDirectory=/usr/share/prometheus
EnvironmentFile=-/etc/conf.d/prometheus
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/data  \
  --web.listen-address="127.0.0.1:9090"
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

# systemd hardening-options
AmbientCapabilities=
CapabilityBoundingSet=
DeviceAllow=/dev/null rw
DevicePolicy=strict
LimitMEMLOCK=0
LimitNOFILE=32768
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
PrivateUsers=true
ProtectControlGroups=true
ProtectHome=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RemoveIPC=true
RestrictNamespaces=true
RestrictRealtime=true
SystemCallArchitectures=native
SyslogIdentifier=prometheus

[Install]
WantedBy=multi-user.target

Run: sudo vim /lib/systemd/system/prometheus-node-exporter.service and insert:

[Unit]
Description=Prometheus exporter for machine metrics
Requires=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Restart=on-failure
EnvironmentFile=-/etc/conf.d/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter --web.listen-address="127.0.0.1:9100" 
NoNewPrivileges=true
ProtectHome=read-only
ProtectSystem=strict

[Install]
WantedBy=multi-user.target

Enable services

# Restart and enable
sudo systemctl daemon-reload

sudo systemctl restart prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

sudo systemctl restart prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

Setup Nginx

# Install
sudo apt install -y nginx

Run: sudo vim /etc/nginx/conf.d/prometheus.conf and insert:

# Configure Nginx as Reverse Proxy

server {
    listen 80;
    server_name <server_ip>; # change here

    location / {
        allow 130.208.240.0/24;  # Current range for Frostbyte VPN
        deny all;

      add_header X-Frame-Options DENY;
      add_header X-Content-Type-Options nosniff;
      add_header X-XSS-Protection "1; mode=block";
      add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

      proxy_pass http://127.0.0.1:9090;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
    }
}

Verify Installation

# Test and restart
sudo nginx -t
sudo systemctl restart nginx

Access Web Interface

• Variables:
  • <server_ip> => The ip of the host machine Prometheus:

# http://<server_ip> # <-- This was in documentation but found out it works on port 9090?
http://<server_ip>:9090

Grafana:

http://<server_ip>:3000

Setup on Clients Guidelines

Debian 12Windows Server 2022

For all VM's to monitor with remote_write to the 'main' monitoring server..

# Create the Prometheus & Node-exporter Users
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

sudo groupadd --system node_exporter
sudo useradd -s /sbin/nologin --system -g node_exporter node_exporter

# verify users
getent passwd prometheus
getent passwd node_exporter

# install 
sudo apt update

cd /tmp

wget https://github.com/prometheus/prometheus/releases/download/v3.1.0/prometheus-3.1.0.linux-amd64.tar.gz
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

tar -xvf prometheus-3.1.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz

sudo mv prometheus-3.1.0.linux-amd64/prometheus /usr/bin/
sudo mv prometheus-3.1.0.linux-amd64/promtool /usr/bin/
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo mkdir /usr/share/prometheus

sudo chown prometheus:prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/share/prometheus
sudo chown prometheus:prometheus /usr/bin/prometheus
sudo chown prometheus:prometheus /usr/bin/promtool

# Verify Services
sudo systemctl is-enabled prometheus
sudo systemctl status prometheus
sudo systemctl is-enabled prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

Configure

Run: sudo vim /etc/prometheus/prometheus.yml and insert:

# Sample config for Prometheus.

global:
scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
    monitor: '$HOSTNAME'

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
    - targets: ['127.0.0.1:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

- job_name: 'node'
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
    - targets: ['127.0.0.1:9100']

remote_write:
- url: 'http://<main-server-ip>:9090/api/v1/write'
    queue_config:
    capacity: 5000
    max_samples_per_send: 100
    batch_send_deadline: 5s

& Run: sudo vim /lib/systemd/system/prometheus.service and insert:

[Unit]
Description=Monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
Requires=network-online.target
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
WorkingDirectory=/usr/share/prometheus
EnvironmentFile=-/etc/conf.d/prometheus
ExecStart=/usr/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/data  \
--web.listen-address="127.0.0.1:9090"
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

# systemd hardening-options
AmbientCapabilities=
CapabilityBoundingSet=
DeviceAllow=/dev/null rw
DevicePolicy=strict
LimitMEMLOCK=0
LimitNOFILE=32768
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
PrivateUsers=true
ProtectControlGroups=true
ProtectHome=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RemoveIPC=true
RestrictNamespaces=true
RestrictRealtime=true
SystemCallArchitectures=native
SyslogIdentifier=prometheus

[Install]
WantedBy=multi-user.target

& Run: sudo vim /lib/systemd/system/prometheus-node-exporter.service and insert:

[Unit]
Description=Prometheus exporter for machine metrics
Requires=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Restart=on-failure
EnvironmentFile=-/etc/conf.d/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter \
    --web.listen-address="127.0.0.1:9100" \
    --collector.processes \
    --collector.meminfo \
    --collector.netstat $NODE_EXPORTER_ARGSNoNewPrivileges=true
NoNewPrivileges=true
ProtectHome=read-only
ProtectSystem=strict

[Install]
WantedBy=multi-user.target

Enable services

# Restart and enable
sudo systemctl daemon-reload

sudo systemctl restart prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

sudo systemctl restart prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

1. [WS22] Download and Install Windows Exporter

• Visit the Windows Exporter Releases.
• Used version in this tutorial: v0.30.4
• Executable used in this tutorial: windows_exporter-0.30.4-amd64.exe

2. [WS22] Configuring Windows Exporter

2.0 - Install Using Either URL [2.1] or Local [2.2] config.yml

2.1 - Install Using URL Configuration

Run the following command in PowerShell:

.\windows_exporter.exe --config.file="https://mus.is/frostbyte/config.yml"

If you need to bypass certificate validation:

.\windows_exporter.exe --config.file="https://mus.is/frostbyte/config.yml" --config.file.insecure-skip-verify

2.2 - Install Using a Local Config File

Create a file called config.yml and add the following example configuration:

# Windows Exporter Configuration Example
collectors:
enabled: cpu,cs,logical_disk,net,os,service,system
collector:
service:
    include: "windows_exporter"
scheduled_task:
    include: /Microsoft/.+
log:
level: debug
scrape:
timeout-margin: 0.5
telemetry:
path: /metrics
max-requests: 5
web:
listen-address: ":9182"

Then, run:

• Run within PowerShell

  .\windows_exporter.exe --config.file="config.yml"
  

• Run in background

  Start-Process -FilePath ".\windows_exporter.exe" -ArgumentList '--config.file="config.yml"' -WindowStyle Hidden
  

3. [WS22] Configure Windows Firewall

Allow inbound traffic on port 9182:

New-NetFirewallRule -DisplayName "Allow Windows Exporter Metrics Port 9182" -Direction Inbound -LocalPort 9182 -Protocol TCP -Action Allow -Profile Any

4. [Prometheus Host] Update Prometheus Configuration

On your Prometheus server, add the Windows target to the bottom of your scrape_config within your prometheus.yml file:

First open the file

sudo vim /etc/prometheus/prometheus.yml

Then add your config, remember to replace the <client-ip> with the IP address of your Windows Server.

scrape_configs:
- job_name: 'windows'
    static_configs:
    - targets: ['<client-ip>:9182']

5. [Prometheus Host] Restart Prometheus

Apply the configuration changes by restarting the Prometheus service:

sudo systemctl restart prometheus.service

6. Verify Connection

Check if the Windows Exporter target is visible in Prometheus:

http://<prometheus-ip>:9090/targets

Replace <prometheus-ip> with the IP of your Prometheus server.

---

Commands to Verify Connectivity

1. Check Exporter Availability

Verify that your Windows exporter is up and being scraped by Prometheus:

up{job="windows"}

Expected Result: A value of 1 indicates that the exporter is reachable.

2. CPU Metrics

To check CPU usage over the last 5 minutes:

rate(windows_cpu_time_total[5m])

For estimating CPU usage (excluding idle time):

100 - (avg by(instance)(irate(windows_cpu_time_total{mode="idle"}[5m])) * 100)

3. Logical Disk Metrics

Check available disk space:

windows_logical_disk_free_bytes

4. Network Metrics

To monitor network traffic over the last 5 minutes:

rate(windows_net_bytes_total[5m])

To group results by network interface:

sum by (if) (rate(windows_net_bytes_total[5m]))

5. OS Information

Retrieve static OS details such as version and build:

windows_os_info