Prometheus and Grafana Monitoring Setup on Ubuntu (Step-by-Step) - Part2

Connecting Prometheus to Grafana & Monitoring Worker Nodes

In this blog, we will learn how to connect Prometheus with Grafana, create dashboards, monitor multiple worker nodes using Node Exporter, generate load for testing, and share Grafana dashboards.

---

🔹 Architecture Overview

• Prometheus → Collects and stores metrics (Time Series Database)

• Node Exporter → Exposes system metrics (CPU, memory, disk, network)

• Grafana → Visualizes metrics using dashboards

• Stress Tool → Generates load for testing

• Loki & Promtail → Log aggregation

• ElasticSearch & Kibana → Log & metric analysis

---

🔹 Step 1: Connect Prometheus to Grafana

1. Login to Grafana Dashboard

   http://<Grafana-Public-IP>:3000
   

2. Navigate to:

   Settings → Data Sources → Add Data Source
   

3. Select Prometheus

4. Enter the Prometheus URL:

   http://<Prometheus-IP>:9090
   

5. Click Save & Test

✔️ If the connection is successful, Grafana will confirm that Prometheus is reachable.

---

🔹 Step 2: Create a Grafana Dashboard

1. On the Grafana top menu, click ➕ (Create)

2. Select Import

3. Enter Dashboard ID:

   1860
   

4. Click Load

5. Choose Prometheus as the data source

6. Click Import

🎉 You will now see a full system monitoring dashboard with CPU, memory, disk, and network metrics.

---

🔹 Step 3: Generate Load for Testing Metrics

To simulate CPU load on the server:

Install stress tool

yum install stress -y

Generate CPU load

stress -c 4

This helps verify whether Prometheus and Grafana are properly collecting and displaying metrics.

---

🔹 Step 4: Connect Worker Nodes to Monitoring Server

1️⃣ Create Worker Nodes

Create 2 worker nodes (EC2 / VM instances).

---

2️⃣ Update /etc/hosts on Monitoring Server

vim /etc/hosts

Add the worker node details:

<public-ip-node1>   worker-1
<public-ip-node2>   worker-2

---

3️⃣ Install Node Exporter on Worker Nodes

📌 Node Exporter is mandatory to collect metrics from worker nodes.

Run the Node Exporter installation script on each worker node.

Once running, verify:

http://<Worker-IP>:9100/metrics

---

🔹 Step 5: Add Worker Nodes to Prometheus

Edit Prometheus configuration:

vim /etc/prometheus/prometheus.yml

Add worker node targets under scrape_configs:

- job_name: "node_exporter"
  static_configs:
    - targets:
        - "worker-1:9100"
        - "worker-2:9100"

---

Restart Prometheus Service

⚠️ Important: After modifying any service configuration, restart the service.

systemctl restart prometheus.service

---

🔹 Step 6: Share Grafana Dashboard

1. Open the dashboard

2. Click Share

3. Select Snapshot

4. Click Create Snapshot

5. Share the generated URL

📌 Note: Replace localhost with public IP before sharing.

Example Snapshot URL:

http://3.80.186.220:3000/dashboard/snapshot/vsz6bR9H8xxj4jawviQsprffK1fFN0Dr

---

🔹 Useful Grafana Dashboard IDs

Dashboard ID	Description
1860	Node Exporter Full
10180	Kubernetes / Node Metrics
14731	Linux System Monitoring
11074	Resource Monitoring
8919	Host Metrics

---

🔹 Monitoring & Logging Tools Summary

Tool	Purpose
Prometheus	Metrics collection
Node Exporter	System metrics
Grafana	Dashboards & visualization
Loki + Promtail	Logs
ElasticSearch	Logs & metrics
Kibana	Dashboards

---

🔹 Conclusion

With Prometheus and Grafana, you can easily monitor multiple servers from a single monitoring node. By adding Node Exporter on worker nodes and updating Prometheus targets, real-time metrics become available instantly. Grafana dashboards help visualize performance, troubleshoot issues, and share insights with teams.