AhFei

AhFei

简洁的写作需要勇气
HomeArchives

Simpler deployment process of Grafana + Prometheus compared to Docker

#linux#self-hosted#probe#Grafana72
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
This article provides a deployment process for Grafana, Prometheus, and Node Exporter, which are popular open-source monitoring tools. The process involves installing and configuring these tools on a Debian-based system, such as Ubuntu. The deployment is estimated to take 15 minutes, but reading the article may also take an additional 15 minutes. The article also mentions that the one-click script provided is only suitable for amd64 systems. Grafana is a visualization program that displays data collected by Prometheus or other data sources. Prometheus is a monitoring and alerting program that polls clients using the HTTP protocol to collect data. Node Exporter is a tool recommended by Prometheus for collecting hardware and kernel metrics. The article provides instructions for deploying and configuring these tools, including opening firewall ports, installing Prometheus and Node Exporter, configuring Prometheus to monitor client nodes, and installing and configuring Grafana to visualize the collected data.

If not specifically stated, follow the order of the commands in the code blocks of the article and execute them one by one to achieve the goal.
Applicable systems: Debian-based distributions, including Ubuntu. Other distributions can also use slightly modified commands in the process.

Estimated deployment time: 15 minutes (but it also takes 15 minutes to read through, so it's 30 minutes?)

The one-click script in the article is only applicable to amd64 systems.

image.png

Grafana, Prometheus, and Node Exporter are a popular set of open-source monitoring tools used to collect, store, query various usage metrics of systems and applications, and visualize them. Each tool has specific functionalities:

  1. Grafana: It is a visualization program that offers both open-source and enterprise versions. Grafana does not collect any data from clients or store it. Its function is to display the data collected by Prometheus or other data sources (such as SQL) in an intuitive and beautiful format. It allows users to customize panels, supports alerts, annotations, dashboard variables, plugins, and authentication.
  2. Prometheus: It is an open-source system monitoring and alerting program that uses HTTP protocol to poll clients for the required data. It maintains a time-series database of the collected results and polls each client at predefined intervals, allowing for an overview of long-term performance of the clients.
  3. Node Exporter: While various collectors can be used on client nodes, Prometheus recommends using its own Node Exporter tool. It collects a wide range of hardware and kernel metrics such as CPU, disk I/O, memory, network, etc. The complete list of available metrics can be found on the https://github.com/prometheus/node_exporter page.

For a better reading experience due to the length of the article, please visit the blog: https://blog.vfly2.com/2023/07/a-simpler-deployment-process-of-the-probe-grafana-prometheus-compared-to-docker/

Resource Usage#

After deploying Grafana and Prometheus, the memory usage increases by more than 200MB. With 3 monitored clients, the memory usage reaches 300MB after running for a day. Overall, it consumes a significant amount of resources and is suitable for monitoring servers with 2GB of memory and 20GB of disk space.

Using the top command, the node_exporter occupies about 20MB of memory.

top -p $(pgrep node_exporter)

Deployment Process#

By default, Grafana web interface is accessed on port 3000, Prometheus on port 9090, and Node Exporter on port 9100. Open the firewall:

sudo ufw allow 9090 comment 'prometheus'
sudo ufw allow 3000 comment 'grafana'
# Node Exporter is different, explained later

For best results, run Prometheus and Grafana on the same server. The server hosting Prometheus and Grafana will be referred to as the "monitoring server", and the systems being monitored will be referred to as "clients".

The steps in this article are written for non-root users. Commands that require elevated privileges are prefixed with sudo.

Firewall on the monitoring server has been opened, completed ✔

If you don't want to use the script or want to understand the installation process, please refer to the original article: https://blog.vfly2.com/2023/07/a-simpler-deployment-process-of-the-probe-grafana-prometheus-compared-to-docker/ (Note: Due to the length of the article, it is arranged this way)

Install Prometheus on the Monitoring Server and Run as a Service#

Copy the one-click installation script for Prometheus: (https://github.com/AhFeil/bash-script/blob/main/install-prometheus.sh)

wget -qO- 'https://raw.githubusercontent.com/AhFeil/bash-script/main/install-prometheus.sh' | sudo bash

If the output shows "active (running)", then the installation is successful.

Access the Prometheus web interface and dashboard by visiting http://ip_addr:9090 in a web browser, replacing ip_addr with the address of the monitoring server.

Install Node Exporter on all Clients and Open the Firewall#

The clients can be Linux distributions or Windows, but Node Exporter is only applicable to Linux.

Open port 9100 on the client only for the monitoring server IP, remember to modify the monitoring server IP:

sudo ufw allow from monitoring_server_IP to any port 9100 comment 'node_exporter'

Copy the one-click installation script for Node Exporter: (https://github.com/AhFeil/bash-script/blob/main/install-node_exporter.sh)

wget -qO- 'https://raw.githubusercontent.com/AhFeil/bash-script/main/install-node_exporter.sh' | sudo bash

If the output shows "active (running)", then the installation is successful.

To update, run the script again.

Configure Prometheus to Monitor the Clients#

The client nodes are now ready for monitoring. To add the clients to prometheus.yml, follow these steps:

On the monitoring server running Prometheus, open prometheus.yml for editing:

sudo vim /etc/prometheus/prometheus.yml

Find scrape_configs, which contains a list of jobs. There is currently one job named prometheus that monitors the local Prometheus task on port 9090.

Below the prometheus job, add a second job with job_name as remote_collector. Include the following information:

  • Set the scrape interval: scrape_interval: 10s
  • Add the IP and port number to monitor, :9100, separating each entry with a comma.
  • To enable monitoring of the local server, add the localhost:9100 entry to the list and install Node Exporter locally.

The entry should be similar to the following example. Replace remote_addr with the actual IP address of the client.

...

  - job_name: "remote_collector"
    scrape_interval: 10s
    static_configs:
      - targets: ["remote_addr:9100", "localhost:9100"]

Refresh Prometheus immediately:

sudo systemctl restart prometheus

Access the Prometheus web portal on port 9090 of the monitoring server using a web browser. Select "Status" - "Targets". The second link for the remote_collector job pointing to port 9100 on the client will be displayed. Click the link to view the statistics.

Install and Deploy Grafana Server#

The statistics collected by Prometheus are just raw data dumps that are difficult to read and not very useful.

Grafana provides an interface to view the statistics collected by Prometheus. Install Grafana on the same server running Prometheus and add Prometheus as a data source.

Preparation:

# Install some necessary utilities
sudo apt-get install -y apt-transport-https software-properties-common
# Import the Grafana GPG key
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
# Add the Grafana "stable version" repository
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Install the open-source version of Grafana:

sudo apt-get update && sudo apt-get install grafana

Reload the systemctl daemon:

sudo systemctl daemon-reload

Enable auto-start on boot and start immediately:

sudo systemctl enable --now grafana-server.service

Check the status:

sudo systemctl status grafana-server

Associate Grafana and Prometheus#

All system components are now installed, but Grafana and Prometheus are not yet associated. The remaining steps can be completed using the Grafana web interface.

To integrate Grafana and Prometheus, follow these steps:

  1. Access the Grafana web interface on port 3000 of the monitoring server in a web browser. For example, enter http://ip_addr:3000, replacing ip_addr with the actual IP address.
  2. Grafana displays the login page. Use the username admin and the default password admin. Change the password to a more secure value when prompted.
  3. After successfully changing the password, Grafana will display the Grafana dashboard.
  4. To add Prometheus as a data source, click the gear icon (representing configuration) and select "Data Sources".

image.png

  1. In the next screen, click the "Add data source" button.
  2. Select Prometheus as the data source.
  3. For the local Prometheus source, set the URL to http://localhost:9090. Most other settings can be left as default.

image.png

  1. Once satisfied with the settings, click the "Save & test" button at the bottom of the screen.
  2. If all settings are correct, Grafana will confirm that the data source is working.

image.png

Use Grafana to Import Dashboard Templates#

You can create custom dashboards. However, Prometheus already created a dashboard to support Node Exporter called "Node Exporter Full". Here is the import process.

To create a custom dashboard, click the dashboard button, which looks like four squares. Then select +New Dashboard. For more information, refer to the Grafana Build a Dashboard guide.

  1. In the Grafana dashboard, select the dashboard icon consisting of four squares,

image.png

  1. Click on the right side 【New】-【Import】.

image.png

  1. Then, in 【Import via grafana.com】, enter the ID 1860 from the previous step. Then select 【Load】.

image.png

  1. Confirm the import details on the next screen. Select Prometheus as the data source, then click the 【Import】 button.

image.png

  1. The "Node Exporter Full" dashboard will be immediately available. It displays performance metrics and status of client nodes, including detailed information about memory, RAM, and CPU.

image.png

Original article link: https://blog.vfly2.com/2023/07/a-simpler-deployment-process-of-the-probe-grafana-prometheus-compared-to-docker/
Copyright: This article is licensed under CC BY-NC-SA 4.0.

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.