Setup Prometheus Clients

Prerequisites

This setup guide is for all Prometheus CLIENTS you want to monitor and expects you to already have the server up and running, if not see here

Install on Clients Guide

For all VM's to monitor with remote_write to the 'main' monitoring server..

# Create the Prometheus & Node-exporter Users
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

sudo groupadd --system node_exporter
sudo useradd -s /sbin/nologin --system -g node_exporter node_exporter

# verify users
getent passwd prometheus
getent passwd node_exporter

# install 
sudo apt update

cd /tmp

wget https://github.com/prometheus/prometheus/releases/download/v3.1.0/prometheus-3.1.0.linux-amd64.tar.gz
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

tar -xvf prometheus-3.1.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz

sudo mv prometheus-3.1.0.linux-amd64/prometheus /usr/bin/
sudo mv prometheus-3.1.0.linux-amd64/promtool /usr/bin/
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo mkdir /usr/share/prometheus

sudo chown prometheus:prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/share/prometheus
sudo chown prometheus:prometheus /usr/bin/prometheus
sudo chown prometheus:prometheus /usr/bin/promtool

# Verify Services
sudo systemctl is-enabled prometheus
sudo systemctl status prometheus
sudo systemctl is-enabled prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

Configure Prometheus

Run: sudo nano /etc/prometheus/prometheus.yml and insert:

Change the script values

Need to change the HOSTNAME and <main-server-ip>

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: '$HOSTNAME' # CHANGE-ME: Uniquely identifies this instance

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['127.0.0.1:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'node'
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['127.0.0.1:9100']

remote_write:
  - url: 'http://<main-server-ip>:9090/api/v1/write' # IP of the monitoring-SERVER
    queue_config:
      capacity: 5000
      max_samples_per_send: 100
      batch_send_deadline: 5s

& Run: sudo nano /lib/systemd/system/prometheus.service and insert:

[Unit]
Description=Monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
Requires=network-online.target
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
WorkingDirectory=/usr/share/prometheus
EnvironmentFile=-/etc/conf.d/prometheus
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/data  \
  --web.listen-address="127.0.0.1:9090"
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

# systemd hardening-options
AmbientCapabilities=
CapabilityBoundingSet=
DeviceAllow=/dev/null rw
DevicePolicy=strict
LimitMEMLOCK=0
LimitNOFILE=32768
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
PrivateUsers=true
ProtectControlGroups=true
ProtectHome=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RemoveIPC=true
RestrictNamespaces=true
RestrictRealtime=true
SystemCallArchitectures=native
SyslogIdentifier=prometheus

[Install]
WantedBy=multi-user.target

& Now run: sudo nano /lib/systemd/system/prometheus-node-exporter.service and insert:

[Unit]
Description=Prometheus exporter for machine metrics
Requires=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Restart=on-failure
EnvironmentFile=-/etc/conf.d/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter --web.listen-address="127.0.0.1:9100" 
NoNewPrivileges=true
ProtectHome=read-only
ProtectSystem=strict

[Install]
WantedBy=multi-user.target

Enable services

# Restart and enable
sudo systemctl daemon-reload

sudo systemctl restart prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

sudo systemctl restart prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

Grafana variables

Edit grafana variable to work with remote_write

Docs can be found here