Description | An open-source Zabbix book |
Author(s) | Patrik Uytterhoeven and all github contributors |
Repository | https://github.com/trikke76/Zabbix-Book |
Copyright | Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License |
Table of Contents
What is this book about ?¶
Hi, welcome and thank you for your interest in my Zabbix book. I wrote the Zabbix cookbook and co wrote with Richards Zabbix 4 Network Monitoring a few years ago for PackPub.
The cookbook the first of it's kind probably outdated and will be replaced by the Zabbix 7 IT Infrastructure Monitoring Cookbook, written by Brian and Nathan, 2 people I like a lot to work with and can higly recommend. There are many more books available from Packt about Zabbix a complete overview can be found here Zabbix books at pack. Or if you like to find some non English books Amazon has some books form Packt and other Publishers in Chinese, Spanish and maybe some other languages as well. Other books
As Zabbix is an opensource product and making money out of the books was never my intention, it got me thinking how to do things different. How to make a new book without using a publisher like I had done before. After a while, I came up with the idea to make a book that would be free and that would be updated when new versions came out. Since I am a huge fan of documentation in markdown or asciidoc I came up with the idea to share the book in git and use markdown. The only problem left was how to make those markdown files readable in an easy way like a book ? After some searching trying to look for a good solution I found MkDocs. MkDocs is a Python-Markdown library that can convert everything to HTML and can be templated. So the problem was solved and a new book was born.
Who am I ?¶
My name is Patrik Uytterhoeven and I work for a Belgium company named Open-Future. I started at this company at January 2013 and that's when my journey started with Zabbix as well. They gave me the opportunity to build my experience and to get certified as Zabbix trainer. Since this year I am officially 10y Zabbix trainer. If you would like to follow one of my trainings feel free to register for a training at our website www.open-future.be. Why would you follow a training if you can read this book for free are you now thinking? Because trainings just like the book explain you all the details on how to set up and do things but also give you valueable tips and feedback that you never get from a book. Books just can't cover everything.
What OS do I need ?¶
Since I work mostly with RHEL based systems and since I am convinced that RHEL is the better choice in Production environments I have chosen to focus on using one of the forks that is available for free. Zabbix is supported on Ubuntu, Debian, Suse, Raspberry .... and it can be compiled on any OS that is Unix based so it's almost impossible to cover them all. However the book is Opensource and in GIT so feel free to contribute the code for your favorite flavour :). I will use Rocky Linux 9 in this book, but it should work for most of the other installations as well.
What version of Zabbix is used in this book ?¶
Since we are almost at the release of Zabbix 7, I will focus on version 7 since it will be the new LTS. It should also apply to most other versions but of course there will be minor changes. In the future, if there is enough support from the community to update this book together, it would be great if we could build a book for every LTS version available.
How to use this book ?¶
The book will try to cover all the topics, feel free to let me know if something is missing or feel free to make a pull request. There is no need to start from page 1 and read the book till the end. Some people will be looking for basic knowledge others might want to skip to the fun part, so I want the book to be useful for everyone. Therefor I will try to explain as best as possible in every topic the exact steps needed to reproduce.
There will be moments in the book where you need to type some code, I will show the commands you need to type in a box just like here.
# some command
Notes to some useful documentation will be added at the bottom of the page.
Here is a simple footnote1. With some additional text after it.
In case there is some important information to share I will add notes in the documentation like can be seen here :
Note
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor massa, nec semper lorem quam in massa.
Info
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor massa, nec semper lorem quam in massa.
Tip
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor massa, nec semper lorem quam in massa.
Question
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor massa, nec semper lorem quam in massa.
Warning
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor massa, nec semper lorem quam in massa.
Bug
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor massa, nec semper lorem quam in massa.
Example
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor massa, nec semper lorem quam in massa.
-
My reference. ↩
Getting started ↵
Requirements¶
Zabbix has a set of requirements that need to be met on the hardware level and software level. These requirements can change over time and also depends on the size of your setup and the software you choose. So before you start buying metal or installing a random database version have a look at the Zabbix documentation and check the latest requirements for the version you want to install. The latest requirements can be found here. Don't forget to select your correct Zabbix version from the list.
If you don't plan to run anything big just a small setup or a test setup Zabbix will run happy on a system with 2cpu and 8G ram. But all depends on how big your setup will be and how many items you will monitor, triggers you will create and for how long you want to keep that data. My advice in the days of Virtualization is you can start small and add more later.
For the setup you can choose to install all components on 1 server or every component on a different server. For the ease of use just make a few notes for yourself:.
server | ip |
---|---|
zabbix server | |
database server | |
web server |
Tip
While zabbix uses dashes "-" in it's names when we need to install packages like zabbix-get or zabbix-sender it's binaries use "_". like zabbix_sender or zabbix_server. This of course can vary depending if you use the packages from the original Zabbix repositories or not. Just be aaware that it's sometimes rather confusing and that if you installed somepackage with a dash that maybe the binary is with an underscore.
Basic OS configuration¶
Firewall¶
It's important for our Zabbix server to have an OS that is well prepared before we start to install our monitoring tool. First we need to make sure our firewall is installed.
# dnf install firewalld --now
Our firewall is installed now, and we are ready to configure the needed ports. For our Zabbix server, we need to allow access to port 10051/tcp this is the port where our Zabbix trapper listens on for incoming data. So we need to open this port in our firewall to allow access to our Zabbix trapper.
# firewall-cmd --add-service=Zabbix-server --permanent
or if the service is not known
# firewall-cmd --add-port=10051/tcp --permanent
firewalld
"Firewalld is the replacement of iptables in Redhat and allows us to make changes available immediately without the need to restart a service. It's possible that your distribution is not using Firewalld in this case you have to look to the documentation of your OS."
Timeserver¶
Another thing we need to configure is the setup of timeserver and sync our Zabbix server to the timeserver by making use of an ntp client. This needs to be done for the Zabbix server but also for the devices we will monitor as time is very important for Zabbix. Imagine one of our hosts having a time zone that is wrong we could end up looking for a problem in Zabbix that happened 6h ago while it had happened maybe only 2h ago.
# dnf install chronyd --now
Chrony should be installed now and enabled and running. This can be verified with the command:
# systemctl status chronyd
dnf
"dnf is a packagemanager from RedHat you need to replace dnf with your correct packagemanager like zyper, apt, yum, ... chrony is a replacement for ntpd and does a better job being faster and more accurate. If your OS does not support chrony then maybe ntpd is still available."
Once Chrony is installed we also need to setup our correct time zone. We can have a look first with 'timedatectl' to see how our time is configured
# timedatectl
Local time: Thu 2023-11-16 15:09:14 UTC
Universal time: Thu 2023-11-16 15:09:14 UTC
RTC time: Thu 2023-11-16 15:09:15
Time zone: UTC (UTC, +0000)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
Make sure that the service cronyd is active, see above on how to do if you missed it. We can choose the correct time zone from a list that we can lookup with the following command:
# timedatectl list-time zones
This will give us a list with all available time zones. Choose the one closest to you.
Africa/Abidjan
Africa/Accra
...
Pacific/Tongatapu
Pacific/Wake
Pacific/Wallis
UTC
We can now configure our correct time zone with the following command:
timedatectl set-time zone Europe/Brussels
When we look again we should see our time zone properly configured.
# timedatectl
Local time: Thu 2023-11-16 16:13:35 CET
Universal time: Thu 2023-11-16 15:13:35 UTC
RTC time: Thu 2023-11-16 15:13:36
Time zone: Europe/Brussels (CET, +0100)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
Note
"Some people like to install all servers in the UTC time zone so that all server logs are in the same time zone when having servers all over the world. Zabbix supports user based time zone settings so it's possible to keep the time zone in UTC on the server and then add the correct time zone in the user interface if you like."
We can test if Chrony is syncronizing with the correct timeservers as well by running the command chronyc
# chronyc
chrony version 4.2
Copyright (C) 1997-2003, 2007, 2009-2021 Richard P. Curnow and others
chrony comes with ABSOLUTELY NO WARRANTY. This is free software, and
you are welcome to redistribute it under certain conditions. See the
GNU General Public License version 2 for details.
chronyc>
Then we type sources
chronyc> sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- 51-15-20-83.rev.poneytel> 2 9 377 354 +429us[ +429us] +/- 342ms
^- 5.255.99.180 2 10 377 620 +7424us[+7424us] +/- 37ms
^- hachi.paina.net 2 10 377 412 +445us[ +445us] +/- 39ms
^* leontp1.office.panq.nl 1 10 377 904 +6806ns[ +171us] +/- 2336us
Here we can see that we are using a bunch of ntp servers that are not in our own country so we better swicht to some timeservers in our local country or if we have a timeserver in our company we could use this one. We can find some local timeservers here : https://www.ntppool.org/
To change this we have to edit our config file "/etc/chrony.conf" and replace the existing ntp server with our local one
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
pool 2.centos.pool.ntp.org iburst
And change it to a local server:
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
pool be.pool.ntp.org iburst
Don't forget to restart the ntpd client of course.
# systemctl restart chronyd
When we look again we will see that we are now using our local timeservers.
chronyc> sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- ntp1.unix-solutions.be 2 6 17 43 -375us[ -676us] +/- 28ms
^* ntp.devrandom.be 2 6 17 43 -579us[ -880us] +/- 2877us
^+ time.cloudflare.com 3 6 17 43 +328us[ +27us] +/- 2620us
^+ time.cloudflare.com 3 6 17 43 +218us[ -83us] +/- 2815us
Installing Zabbix DB Server¶
In this chapter we will install our Zabbix database. Zabbix support a wide range of SQL databases but we will limit us to PostgreSQL, MariaDB and MySQL for now. There is also Oracle that is supported but support for Oracle is deprectated, Zabbix 7 will be the last version of Zabbix supporting Oracle as backend. In this Chapter we will explain how to install and configure every database on a seperate machine however you could also install it on your Zabbix server, there is no rule that forces you to install the DB on your Zabbix server or on a seperate server.
For most setups a DB on a local machine will probably be enough to start with, just make sure the DB is on other disks then the OS. If later performance is an issue you can still move the DB to another server. Don't forget that local connections are faster then connections over TCP so there it's not always best to move the DB to it's own server it all depends on your needs.
Installing Zabbix with MariaDB¶
Let us start with the installation of the MariaDB server, you need to create a MariaDB repository configuration file mariadb.repo
manually in the following path /etc/yum.repos.d/
.
To create a MariaDB repository file, you can use the following command.
Add the MariaDB repo¶
# vi /etc/yum.repos.d/mariadb.repo
The above command will create a new repository file, Once it is created, you need to add the following configuration into the file. Make sure your version, in this case 10.11, is supported by Zabbix by looking at the latest requirements for your version.
# MariaDB 10.11 RedHatEnterpriseLinux repository list - created 2023-11-01 14:20 UTC
# https://mariadb.org/download/
[mariadb]
name = MariaDB
# rpm.mariadb.org is a dynamic mirror if your preferred mirror goes offline. See https://mariadb.org/mirrorbits/ for details.
# baseurl = https://rpm.mariadb.org/10.11/rhel/$releasever/$basearch
baseurl = https://mirror.23m.com/mariadb/yum/10.11/rhel/$releasever/$basearch
# gpgkey = https://rpm.mariadb.org/RPM-GPG-KEY-MariaDB
gpgkey = https://mirror.23m.com/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck = 1
Lets update our OS first with the latest patches
# dnf update -y
Install the MariaDB database¶
Now we are ready to install our MariaDB database.
# dnf install MariaDB-server MariaDB-client
We are now ready to enable and start or MariaDB database.
# systemctl enable mariadb --now
Once the installation is complete, you can verify the version of the MariaDB server by using the following command:
# mysql -V
The output should look like this:
mysql Ver 15.1 Distrib 10.11.6-MariaDB, for Linux (x86_64) using EditLine wrapper
And when we ask the status of our MariaDB server we should get an output like this:
# systemctl status mariadb
● mariadb.service - MariaDB 10.11.6 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Sat 2023-11-18 19:19:36 CET; 2min 13s ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 41986 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 41987 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START>
Process: 42006 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Main PID: 41995 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 9 (limit: 12344)
Memory: 206.8M
CPU: 187ms
Securing the MariaDB database¶
It's time to secure our database by removing the test database and user and set our own root password. Run the command mariadb-secure-installation
, you should get the following output.
# mariadb-secure-installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.
You already have your root account protected, so you can safely answer 'n'.
Switch to unix_socket authentication [Y/n] n
... skipping.
You already have your root account protected, so you can safely answer 'n'.
Change the root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] y
... Success!
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
Create the Zabbix database¶
# mysql -uroot -p
password
MariaDB [(none)]> CREATE DATABASE zabbix CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
MariaDB [(none)]> CREATE USER 'zabbix-web'@'<zabbix server ip>' IDENTIFIED BY '<password>';
MariaDB [(none)]> CREATE USER 'zabbix-srv'@'<zabbix server ip>' IDENTIFIED BY '<password>';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix-srv'@'<zabbix server ip>';
MariaDB [(none)]> GRANT SELECT, UPDATE, DELETE, INSERT ON zabbix.* TO 'zabbix-web'@'<zabbix server ip>';
MariaDB [(none)]> SET GLOBAL log_bin_trust_function_creators = 1;
MariaDB [(none)]> QUIT
Warning
"The Zabbix documentation explicitly mentions that deterministic triggers need to be created during the import of schema. On MySQL and MariaDB, this requires GLOBAL log_bin_trust_function_creators = 1 to be set if binary logging is enabled and there is no superuser privileges and log_bin_trust_function_creators = 1 is not set in MySQL configuration file."
Add the Zabbix repository and populate the DB¶
# rpm -Uvh https://repo.zabbix.com/zabbix/6.5/rocky/9/x86_64/zabbix-release-6.5-2.el9.noarch.rpm
# dnf clean all
# dnf install zabbix-sql-scripts
Upload the data from zabbix (db structure, images, user, ... )
# zcat /usr/share/zabbix-sql-scripts/mysql/server.sql.gz | mysql --default-character-set=utf8mb4 -uroot -p zabbix
Warning
"Depending on the speed of your hardware or VM this can take a few seconds upto a few minutes so please don't cancel just sit and wait for the prompt."
Log back into your MariaDB Database as root
# mysql -uroot -p
Remove the global parameter again as its not needed anymore and also for security reasons.
MariaDB [(none)]> SET GLOBAL log_bin_trust_function_creators = 0;
Query OK, 0 rows affected (0.001 sec)
Configure the firewall¶
One last thing we need to do is open the firewall and allow incoming connections for the MariaDB database from our Zabbix server because at the moment we dont accept any connections yet.
# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp0s3 enp0s8
sources:
services: cockpit dhcpv6-client ssh
ports:
protocols:
forward: yes
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
First we will create an appropriate zone for our MariaDB and open port 3306/tcp but only for the ip from our Zabbix server.
# firewall-cmd --new-zone=mariadb-access --permanent
success
# firewall-cmd --reload
success
# firewall-cmd --get-zones
block dmz drop external home internal mariadb-access nm-shared public trusted work
# firewall-cmd --zone=mariadb-access --add-source=<zabbix-serverip> --permanent
success
# firewall-cmd --zone=mariadb-access --add-port=3306/tcp --permanent
success
# firewall-cmd --reload
Now lets have a look to our firewall rules to see if they are what we expected:
# firewall-cmd --zone=mariadb-access --list-all
mariadb-access (active)
target: default
icmp-block-inversion: no
interfaces:
sources: <ip from zabbix-server>
services:
ports: 3306/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
Our database server is ready now to accept connections from our Zabbix server :). You can continue with the next task Installing the Zabbix Server
Installing Zabbix with MySQL¶
Let us start with the installation of the MySQL server, you need to create a MySQL repository first so that we can install the proper files for our MySQL server It's alwqys best to check the Zabbix documentation to see what version is supported so you don't install a version that is not supported or is not supported anymore.
Add the MySQL repo¶
Run the following command to install the MySQL repo for version 8.0
# dnf -y install https://dev.mysql.com/get/mysql80-community-release-el9-1.noarch.rpm
Note
"If you install this on RedHat 8 and higher or alternatives like CentOS, Rocky or Alma 8 then you need to disable the mysql module by running 'module disable mysql'."
Let's update our OS first with the latest patches
# dnf update -y
Installing the MySQL database¶
# dnf -y install mysql-community-server
We are now ready to enable and start or MySQL database.
# systemctl enable mysqld --now
Once the installation is complete, you can verify the version of the MySQL server by using the following command:
# mysql -V
The output should look like this:
mysql Ver 8.0.35 for Linux on x86_64 (MySQL Community Server - GPL)
And when we ask the status of our MariaDB server we should get an output like this:
# systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; preset: disabled)
Active: active (running) since Mon 2023-11-20 22:15:51 CET; 1min 15s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 44947 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 45012 (mysqld)
Status: "Server is operational"
Tasks: 37 (limit: 12344)
Memory: 448.3M
CPU: 4.073s
CGroup: /system.slice/mysqld.service
└─45012 /usr/sbin/mysqld
Nov 20 22:15:43 mysql-db systemd[1]: Starting MySQL Server...
Nov 20 22:15:51 mysql-db systemd[1]: Started MySQL Server.
Securing the MySQL database¶
MySQL will secure our database with a random root password that is generated when we install the database. First thing we need to do is replace it with our own password. To find what the password is we need to read the log file with the followin command:
# grep 'temporary password' /var/log/mysqld.log
Change the root password as soon as possible by logging in with the generated, temporary password and set a custom password for the superuser account:
# mysql -uroot -p
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '<my mysql password>';
mysql> quit
Next we can run the command mysql_secure_installation, you should get the following output:
Note
"There is no need to reset the root password for MySQL again as we have reset it already. The next step is optional but recommended."
# mysql_secure_installation
Securing the MySQL server deployment.
Enter password for user root:
The 'validate_password' component is installed on the server.
The subsequent steps will run with the existing configuration
of the component.
Using existing password for root.
Estimated strength of the password: 100
Change the password for root ? ((Press y|Y for Yes, any other key for No) : n
... skipping.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
- Dropping test database...
Success.
- Removing privileges on test database...
Success.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.
All done!
Let's create our DB users and the correct permissions in the database:
mysql -uroot -p
mysql> CREATE DATABASE zabbix CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
mysql> CREATE USER 'zabbix-web'@'<zabbix server ip>' IDENTIFIED BY '<password>';
mysql> CREATE USER 'zabbix-srv'@'<zabbix server ip>' IDENTIFIED BY '<password>';
mysql> GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix-srv'@'<zabbix server ip>';
mysql> GRANT SELECT, UPDATE, DELETE, INSERT ON zabbix.* TO 'zabbix-web'@'<zabbix server ip>';
mysql> SET GLOBAL log_bin_trust_function_creators = 1;
mysql> QUIT
Warning
"The Zabbix documentation explicitly mentions that deterministic triggers need to be created during the import of schema. On MySQL and MariaDB, this requires GLOBAL log_bin_trust_function_creators = 1 to be set if binary logging is enabled and there is no superuser privileges and log_bin_trust_function_creators = 1 is not set in MySQL configuration file."
Add the Zabbix repository and populate the DB¶
# rpm -Uvh https://repo.zabbix.com/zabbix/6.5/rocky/9/x86_64/zabbix-release-6.5-2.el9.noarch.rpm
# dnf clean all
# dnf install zabbix-sql-scripts
Now let;s upload the data from zabbix (db structure, images, user, ... )
# zcat /usr/share/zabbix-sql-scripts/mysql/server.sql.gz | mysql --default-character-set=utf8mb4 -uroot -p zabbix
Enter password:
Warning
"Depending on the speed of your hardware or VM this can take a few seconds upto a few minutes so please don't cancel just sit and wait for the prompt."
Log back into your MySQL Database as root
# mysql -uroot -p
Remove the global parameter again as its not needed anymore and also for security reasons.
mysql> SET GLOBAL log_bin_trust_function_creators = 0;
Query OK, 0 rows affected, 1 warning (0.00 sec)
Configure the firewall¶
One last thing we need to do is open the firewall and allow incoming connections from our Zabbix server to our MySQL database because at the moment we dont accept any connections yet.
# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp0s3 enp0s8
sources:
services: cockpit dhcpv6-client ssh
ports:
protocols:
forward: yes
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
First we will create an appropriate zone for our MySQL Database and open port 3306/tcp but only for the IP from our Zabbix server. This way no one unallowed is able to connect.
# firewall-cmd --new-zone=mysql-access --permanent
success
# firewall-cmd --reload
success
# firewall-cmd --get-zones
block dmz drop external home internal mysql-access nm-shared public trusted work
# firewall-cmd --zone=mysql-access --add-source=<zabbix-serverip> --permanent
success
# firewall-cmd --zone=mysql-access --add-port=3306/tcp --permanent
success
# firewall-cmd --reload
Now lets have a look to our firewall rules to see if they are what we expected:
# firewall-cmd --list-all --zone=mysql-access
mysql-access (active)
target: default
icmp-block-inversion: no
interfaces:
sources: <ip from the zabbix-server>
services:
ports: 3306/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
Our database server is ready now to accept connections from our Zabbix server :). You can continue with the next task Installing the Zabbix Server
Installing Zabbix with PostgreSQL¶
For our DB setup with PostgreSQL we need to add our PostgreSQL repository first to the system. As of writing PostgreSQL 13-16 are supported but best is to have a look before you install it as new versions may be supported and older maybe unsupported both by Zabbix and PostgreSQL. Usually it's a good idea to go with the latest version that is supported by Zabbix. Zabbix also supports the extension TimescaleDB this is someting we will talk later about. As you will see the setup from PostgreSQL is very different from MySQL not only the installation but also securing the DB.
The table of compatibility can be found here.
Add the PostgreSQL repo¶
So let us start first setting up our PostgreSQL repository with the folowing commands.
# Install the repository RPM:
sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
# Disable the built-in PostgreSQL module:
sudo dnf -qy module disable postgresql
# Install PostgreSQL:
sudo dnf install -y postgresql16-server
# Initialize the database and enable automatic start:
sudo /usr/pgsql-16/bin/postgresql-16-setup initdb
sudo systemctl enable postgresql-16 --now
Securing the PostgreSQL database¶
As i told you PostgreSQL works a bit different then MySQL or MariaDB and this applies aswell to how we manage access permissions. Postgres works with a file with the name pg_hba.conf where we have to tell who can access our database from where and what encryption is used for the password. So let's edit this file to allow our frontend and zabbix server to access the database.
Note
"Client authentication is configured by a configuration file with the name pg_hba.conf
. HBA here stands for host based authentication. For more information feel free to check the PostgreSQL documentation."
Add the following lines, the order here is important.
# vi /var/lib/pgsql/16/data/pg_hba.conf
# "local" is for Unix domain socket connections only
local zabbix zabbix-srv scram-sha-256
local all all peer
# IPv4 local connections:
host zabbix zabbix-srv <ip from zabbix server/24> scram-sha-256
host zabbix zabbix-web <ip from zabbix server/24> scram-sha-256
host all all 127.0.0.1/32 scram-sha-256
After we changed the pg_hba file don't forget to restart postgres else the settings will not be applied. But before we restart let us also edit the file postgresql.conf and allow our database to listen on our network interface for incomming connections from the zabbix server. Postgresql will standard only allow connections from the socket.
# vi /var/lib/pgsql/16/data/postgresql.conf
and replace the line with listen_addresses so that PostgreSQL will listen on all interfaces and not only on our localhost.
#listen_addresses = 'localhost' with listen_addresses = '*'
When done restart the PostgreSQL cluster and see if it comes back online in case of an error check the pg_hba.conf
file you just edited for typos.
# systemctl restart postgresql-16
For our Zabbix server we need to create tables in the database for this we need ot install the Zabbix repository like we did for our Zabbix server and install the Zabbix package containing all the database tables images icons, ....
Add the Zabbix repository and populate the DB¶
# dnf install https://repo.zabbix.com/zabbix/6.0/rhel/9/x86_64/zabbix-release-6.0-4.el9.noarch.rpm -y
# dnf install zabbix-sql-scripts -y
Now we are ready to create our Zabbix users for the server and the frontend:
# su - postgres
# createuser --pwprompt zabbix-srv
Enter password for new role: <server-password>
Enter it again: <server-password>
Let's do the same for our frontend let's create a user to connect to the database:
# createuser --pwprompt zabbix-web
Enter password for new role: <frontend-password>
Enter it again: <frontend-password>
Next we have to unzip the database schema files. Run as user root followin command::
# gzip -d /usr/share/zabbix-sql-scripts/postgresql/server.sql.gz
We are now ready to create our database zabbix. Become user postgres again and run next command to create the database as our user zabbix-srv:
# su - postgres
# createdb -E Unicode -O zabbix-srv zabbix
Let's verify that we are really connected to the database with the correct session. Login from the Postgres shell on the zabbix database
# psql -d zabbix -U zabbix-srv
Make sure we are logged in with our correct user zabbix-srv
.
zabbix=> SELECT session_user, current_user;
session_user | current_user
--------------+--------------
zabbix-srv | zabbix-srv
(1 row)
PostgreSQL works a bit different then MySQL or MariaDB when it comes to almost everything :) One of the things that it has that MySQL not has are for example shemas. If you like to know more about it i can recommend this URI. It explains in detail what it is and why we need it. But in short ... In PostgreSQL schema enables a multi-user environment that allows multiple users to access the same database without interference. Schemas are important when several users use the application and access the database in their way or when various applications utilize the same database. There is a standard schema that you can use but the better way is to create our own schema.
Note
"There is a standard schema public
that you can use but the better way is to create our own schema this was if later something else is installed next to the Zabbix database it will be easier to create users with only access to the newly created database tables."
zabbix=> CREATE SCHEMA zabbix_server AUTHORIZATION "zabbix-srv";
CREATE SCHEMA
zabbix=> set search_path to "zabbix_server";
zabbix=> \dn
List of schemas
Name | Owner
---------------+-------------------
public | pg_database_owner
zabbix_server | zabbix-srv
(2 rows)
Now we have our DB ready with correct permissions for user zabbix-srv
but not yet for our user zabbix-web
. Let's fix this first and give the rights to connect to our schema.
zabbix=# GRANT USAGE ON SCHEMA zabbix_server TO "zabbix-web";
GRANT
The user zabbix-web
has now the rights to connect to our schema but cannot to anything yet lets fix this but also don't give too many rights.
zabbix=# GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA zabbix_server TO "zabbix-web";
GRANT
zabbix=# GRANT SELECT, UPDATE ON ALL SEQUENCES IN SCHEMA zabbix_server TO "zabbix-web";
GRANT
There we go both users are created with the correct permissons. We are now ready to populate the database with the Zabbix table structures etc ... log back in as user postgres and run the following commands
Let's upload the Zabbix SQL file we extracted earlier to populate our database with the needed schemas images users etc ...
Warning
"Depending on the speed of your hardware or VM this can take a few seconds upto a few minutes so please don't cancel just sit and wait for the prompt."
zabbix=# \i /usr/share/zabbix-sql-scripts/postgresql/server.sql
CREATE TABLE
CREATE INDEX
...
...
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
COMMIT
zabbix=#
Note
"If the import fails with psql:/usr/share/zabbix-sql-scripts/postgresql/server.sql:7: ERROR: no schema has been selected to create in
then you probably made an error in the line where you set the search path."
Lets verify that our tables are properly created with the correct permissions
zabbix=# \dt
List of relations
Schema | Name | Type | Owner
---------------+----------------------------+-------+------------
zabbix_server | acknowledges | table | zabbix-srv
zabbix_server | actions | table | zabbix-srv
zabbix_server | alerts | table | zabbix-srv
zabbix_server | auditlog | table | zabbix-srv
zabbix_server | autoreg_host | table | zabbix-srv
...
...
zabbix_server | usrgrp | table | zabbix-srv
zabbix_server | valuemap | table | zabbix-srv
zabbix_server | valuemap_mapping | table | zabbix-srv
zabbix_server | widget | table | zabbix-srv
zabbix_server | widget_field | table | zabbix-srv
(173 rows)
Note
"If you are like me and don't like to set the search path every time you logon with the user zabbix-srv to the correct search path you can run the following SQL. zabbix=> alter role "zabbix-srv" set search_path = "$user", public, zabbix_server ;
"
If you are ready you can exit the database and return as user root.
zabbix=> \q
# exit
Configure the firewall¶
One last thing we need to do is open the firewall and allow incoming connections for the PostgreSQL database from our Zabbix server because at the moment we dont accept any connections yet.
# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp0s3 enp0s8
sources:
services: cockpit dhcpv6-client ssh
ports:
protocols:
forward: yes
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
First we will create an appropriate zone for our PostgreSQL DB and open port 5432/tcp but only for the ip from our Zabbix server.
# firewall-cmd --new-zone=postgresql-access --permanent
success
# firewall-cmd --reload
success
# firewall-cmd --get-zones
block dmz drop external home internal nm-shared postgresql-access public trusted work
# firewall-cmd --zone=postgresql-access--add-source=<zabbix-serverip> --permanent
success
# firewall-cmd --zone=postgresql-access --add-port=5432/tcp --permanent
success
# firewall-cmd --reload
Now lets have a look to our firewall rules to see if they are what we expected:
# firewall-cmd --zone=postgresql-access --list-all
postgresql-access (active)
target: default
icmp-block-inversion: no
interfaces:
sources: 192.168.56.18
services:
ports: 5432/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
Our database server is ready now to accept connections from our Zabbix server :). You can continue with the next task Installing the Zabbix Server
Installing Zabbix¶
In this chapter we will cover the installation of our Zabbix server. In the previous topic we already installed the DB now we will see how to install our Zabbix server and to connect it to our database. We will cover also possible design choices as Zabbix can be installed in a modular way.
Before we can install Zabbix we first have to know how the design is. The Zabbix server has been build op modular based on 3 components.
- The Zabbix server
- The Zabbix web server
- The Zabbix database
All these components can be installed on 1 server or can be split over 3 different servers. The Zabbix server itself is the brain this part is doing all the trigger calculations and sending all the alert. The database is where the Zabbix server stores its config and all the data that we have gathered. The web server provides us with a front-end. Note that Zabbix has a API and that this is also located on the front-end and not on the Zabbix server side.
All these parts have to work together so as you can see in our image above. The Zabbix server needs to read the config and store the data in our database and the Zabbix front-end needs to be able to write the configuration in the database as well. The Zabbix front-end also needs to check the online status of our Zabbix server and needs to read some other information as well.
For our setup, we will use 2 VM's, 1 VM with a Zabbix server and our Zabbix web server and another VM with our Zabbix database.
In this chapter we will see how to :
- Install the Zabbix server for MySQL/MariaDB
- Install the Zabbix server for PostgreSQL
- Install the Zabbix frontend with NginX
- Install the Zabbix frontend with Apache
Installing the Zabbix Server¶
Before you start to install your Zabbix server make sure the server is properly configure as we explained in our topic Basic OS configuration before we start.
Something else that is important in this case is that we need to disable SELinux. We will see later in chapter Securing Zabbix with SELinux how to do this properly.
We can check the status of SELinux with the command sestatus
:
# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
As you can see we are now in enforcing mode.
To disable SELinux just run setenforce 0
to disable it.
# setenforce 0
# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
As you can see our current mode is now permissive.
However this is not persistent so we also need to alter our SELinux configuration file. This can be done by altering the file /etc/config/selinux
and replacing enforcing by permissive.
A more easy way is to run the following command :
sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/selinux/config
This line will alter the config file for you. So when we run sestatus
again we will see that we are in permissive
mode and that our config file is also in permissive
mode.
We can verify this with our cat commando.
# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
# See also:
# https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-selinux/#getting-started-with-selinux-selinux-states-and-modes
#
# NOTE: In earlier Fedora kernel builds, SELINUX=disabled would also
# fully disable SELinux during boot. If you need a system with SELinux
# fully disabled instead of SELinux running with no policy loaded, you
# need to pass selinux=0 to the kernel command line. You can use grubby
# to persistently set the bootloader to boot with selinux=0:
#
# grubby --update-kernel ALL --args selinux=0
#
# To revert back to SELinux enabled:
#
# grubby --update-kernel ALL --remove-args selinux
#
SELINUX=permissive
# SELINUXTYPE= can take one of these three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
And we can also verify it with our commando setstatus
# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
Adding the Zabbix repository¶
From the Zabbix Download page select the correct Zabbix version you would like to install. In our case it will be 7.0 LTS. Select the correct OS distribution as well. This will be Rocky Linux 9 in our case. We are going to install the Server and will be using NGINX.
Our first step is to disable Zabbix packages provided by EPEL, if you have it installed. Edit file /etc/yum.repos.d/epel.repo and add the following statement.
[epel]
...
excludepkgs=zabbix*
Tip
Having the EPEL repository enabled is a bad practice and could be dangerous if you use EPEL it's best to disable the repo and use dnf install
Our next task is to install the Zabbix repository on our OS and do a dnf cleanup so that old cache files from our repository metadata is cleaned up.
rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rhel/9/x86_64/zabbix-release-6.5-2.el9.noarch.rpm
dnf clean all
Note
A repository is a config in Linux that you can add to make packages available for you OS to install. The best way to look at it is maybe to think of it like an APP store that you add where you can find the software of your vendor. In this case the repository form Zabbix. There are many repositories you can add but you should be sure that they can be trusted. So it's always a good idea to stick to the repositories of your OS and only add extra repositories when you are sure they are to be trusted and needed. In our case the repository is from our vendor Zabbix so it should be safe to add. Epel is another popular repository for RedHat systems that is considered to be safe.
Installing the Zabbix server for MySQL/MariaDB¶
Now that we have our repository with software added to our system we are ready to install our Zabbix server and webserver. Remember the webserver could be installed on another system. There is no need to install both on the same server.
dnf install zabbix-server-mysql zabbix-web-mysql
Now that we have installed our packages for the Zabbix server and our frontend we still need to change the configuration of our Zabbix server so that we can connect to our database. Open the file /etc/zabbix/zabbix_server.conf
and replace the following lines:
DBHost=<ip or dns of your MariaDB server>
DBName=<the name of your database>
DBUser=<the user that will connect to the database>
DBPassword=<your super secret password>
Make sure you don't have a '#' in front of the config parameter else Zabbix will see this as text and not as a parameter. Also make sure that there are not extra duplicate lines Zabbix will always take the last config parameter if there is more then 1 line with the same parameter
In our case the config will look like this:
# vi /etc/zabbix/zabbix_server.conf
DBHost=<ip or dns of your MariaDB server>
DBName=zabbix
DBUser=zabbix-srv
DBPassword=<your super secret password>
DBPort=3306
Note
The Zabbix server configuration file has the option to include an extra config file with parameters you like to alter or add. In production it's probably better to not touch the configuration file but to add a new file and include the parameters you like to change. This way you never have to edit your original configuration file after an upgrade it will also make your life more easy when working with configuration tools like Ansible, Puppet, SaltStack, .... The only thing that needs to be done is remove the # in front of the line '# Include=/usr/local/etc/zabbix_server.conf.d/*.conf' and make sure the path exists with a customized config file of your won that is readable by the user zabbix.
Ok now that we have changed the configuration of you Zabbix server so that it is able to connect to our DB we are ready to start. Run the following command to enable the Zabbix server and make it active on boot next time.
systemctl enable zabbix-server --now
Our Zabbix server service will start and if everything goes well you should see in the Zabbix server log file the following output
tail /var/log/zabbix/zabbix_server.log
1123:20231120:110604.440 Starting Zabbix Server. Zabbix 7.0.0alpha7 (revision 60de6a81aca).
1123:20231120:110604.440 ****** Enabled features ******
1123:20231120:110604.440 SNMP monitoring: YES
1123:20231120:110604.440 IPMI monitoring: YES
1123:20231120:110604.440 Web monitoring: YES
1123:20231120:110604.440 VMware monitoring: YES
1123:20231120:110604.440 SMTP authentication: YES
1123:20231120:110604.440 ODBC: YES
1123:20231120:110604.440 SSH support: YES
1123:20231120:110604.440 IPv6 support: YES
1123:20231120:110604.440 TLS support: YES
1123:20231120:110604.440 ******************************
1123:20231120:110604.440 using configuration file: /etc/zabbix/zabbix_server.conf
1123:20231120:110604.470 current database version (mandatory/optional): 06050143/06050143
1123:20231120:110604.470 required mandatory version: 06050143
1124:20231120:110604.490 starting HA manager
1124:20231120:110604.507 HA manager started in active mode
1123:20231120:110604.508 server #0 started [main process]
1126:20231120:110604.509 server #2 started [configuration syncer #1]
1125:20231120:110604.510 server #1 started [service manager #1]
1133:20231120:110604.841 server #9 started [lld worker #1]
1132:20231120:110604.841 server #8 started [lld manager #1]
1134:20231120:110604.841 server #10 started [lld worker #2]
If there was an error and the server was not able to connect to the database you would see something like this in the server log file :
10773:20231118:213248.570 Starting Zabbix Server. Zabbix 7.0.0alpha7 (revision 60de6a81aca).
10773:20231118:213248.570 ****** Enabled features ******
10773:20231118:213248.570 SNMP monitoring: YES
10773:20231118:213248.570 IPMI monitoring: YES
10773:20231118:213248.570 Web monitoring: YES
10773:20231118:213248.570 VMware monitoring: YES
10773:20231118:213248.570 SMTP authentication: YES
10773:20231118:213248.570 ODBC: YES
10773:20231118:213248.570 SSH support: YES
10773:20231118:213248.570 IPv6 support: YES
10773:20231118:213248.570 TLS support: YES
10773:20231118:213248.570 ******************************
10773:20231118:213248.570 using configuration file: /etc/zabbix/zabbix_server.conf
10773:20231118:213248.574 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to server on 'xxx.xxx.xxx.xxx' (115)
10773:20231118:213248.574 database is down: reconnecting in 10 seconds
10773:20231118:213258.579 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to server on 'xxx.xxx.xxx.xxx' (115)
10773:20231118:213258.579 database is down: reconnecting in 10 seconds
Let's check the Zabbix server service to see if it's enabled so that it survives a reboot
# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; preset: disabled)
Active: active (running) since Mon 2023-11-20 11:06:04 CET; 1h 2min ago
Main PID: 1123 (zabbix_server)
Tasks: 59 (limit: 12344)
Memory: 52.6M
CPU: 20.399s
CGroup: /system.slice/zabbix-server.service
├─1123 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
├─1124 "/usr/sbin/zabbix_server: ha manager"
├─1125 "/usr/sbin/zabbix_server: service manager #1 [processed 0 events, updated 0 event tags, deleted 0 problems, synced 0 service updates, idle 5.008686 sec during 5.016382 sec]"
├─1126 "/usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.092797 sec, idle 10 sec]"
├─1127 "/usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.027620 sec during 5.027828 sec]"
├─1128 "/usr/sbin/zabbix_server: alerter #1 started"
├─1129 "/usr/sbin/zabbix_server: alerter #2 started"
├─1130 "/usr/sbin/zabbix_server: alerter #3 started"
├─1131 "/usr/sbin/zabbix_server: preprocessing manager #1 [queued 1, processed 2 values, idle 5.490312 sec during 5.490555 sec]"
├─1132 "/usr/sbin/zabbix_server: lld manager #1 [processed 1 LLD rules, idle 5.028973sec during 5.029123 sec]"
├─1133 "/usr/sbin/zabbix_server: lld worker #1 [processed 1 LLD rules, idle 60.060180 sec during 60.085009 sec]"
├─1134 "/usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 60.065526 sec during 60.095165 sec]"
├─1135 "/usr/sbin/zabbix_server: housekeeper [deleted 0 hist/trends, 0 items/triggers, 0 events, 0 sessions, 0 alarms, 0 audit items, 0 autoreg_host, 0 records in 0.019108 sec, idle for 1 hour(s)]"
├─1136 "/usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.002856 sec, idle 59 sec]"
├─1137 "/usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000059 sec, idle 5 sec]"
├─1138 "/usr/sbin/zabbix_server: discovery manager #1 [processing 0 rules, 0.000000% of queue used, 0 unsaved checks]"
├─1139 "/usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000036 sec, idle 1 sec]"
├─1140 "/usr/sbin/zabbix_server: history syncer #2 [processed 1 values, 0 triggers in 0.005016 sec, idle 1 sec]"
├─1141 "/usr/sbin/zabbix_server: history syncer #3 [processed 0 values, 0 triggers in 0.000031 sec, idle 1 sec]"
├─1142 "/usr/sbin/zabbix_server: history syncer #4 [processed 0 values, 0 triggers in 0.000014 sec, idle 1 sec]"
├─1143 "/usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.005587 sec, idle 3 sec]"
├─1144 "/usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000010 sec, idle 5 sec]"
├─1145 "/usr/sbin/zabbix_server: self-monitoring [processed data in 0.000016 sec, idle 1 sec]"
├─1146 "/usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.002511 sec, idle 5 sec]"
├─1147 "/usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000009 sec, idle 1 sec]"
├─1148 "/usr/sbin/zabbix_server: poller #2 [got 1 values in 0.000232 sec, idle 1 sec]"
├─1149 "/usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000015 sec, idle 1 sec]"
├─1150 "/usr/sbin/zabbix_server: poller #4 [got 0 values in 0.000010 sec, idle 1 sec]"
This concludes our chapter on installing and configuring our Zabbix server. Next we have to configure our frontend. You can have a look at Installing Zabbix frontend with Nginx or Installing Zabbix frontend with Apache
Installing the Zabbix server for PostgreSQL¶
Now that we have our repository with software added to our system we are ready to install our Zabbix server and webserver. Remember the webserver could be installed on another system. There is no need to install both on the same server.
dnf install zabbix-server-pgsql zabbix-web-pgsql
Now that we have installed our packages for the Zabbix server and our frontend we still need to change the configuration of our Zabbix server so that we can connect to our database. Open the file /etc/zabbix/zabbix_server.conf
and replace the following lines:
DBHost=<ip or dns of your PostgreSQL server>
DBName=<the name of your database>
DBSchema=<our PostgreSQL schema name>
DBUser=<the user that will connect to the database>
DBPassword=<your super secret password>
Make sure you don't have a '#' in front of the config parameter else Zabbix will see this as text and not as a parameter. Also make sure that there are not extra duplicate lines Zabbix will always take the last config parameter if there is more then 1 line with the same parameter
In our case the config will look like this:
# vi /etc/zabbix/zabbix_server.conf
DBHost=<ip or dns of your MariaDB server>
DBName=zabbix
DBSchema=zabbix_server
DBUser=zabbix-srv
DBPassword=<your super secret password>
DBPort=5432
Note
The Zabbix server configuration file has the option to include an extra config file with parameters you like to alter or add. In production it's probably better to not touch the configuration file but to add a new file and include the parameters you like to change. This way you never have to edit your original configuration file after an upgrade it will also make your life more easy when working with configuration tools like Ansible, Puppet, SaltStack, .... The only thing that needs to be done is remove the # in front of the line '# Include=/usr/local/etc/zabbix_server.conf.d/*.conf' and make sure the path exists with a customized config file of your won that is readable by the user zabbix.
Ok now that we have changed the configuration of you Zabbix server so that it is able to connect to our DB we are ready to start. Run the following command to enable the Zabbix server and make it active on boot next time.
systemctl enable zabbix-server --now
Our Zabbix server service will start and if everything goes well you should see in the Zabbix server log file the following output
tail /var/log/zabbix/zabbix_server.log
1123:20231120:110604.440 Starting Zabbix Server. Zabbix 7.0.0alpha7 (revision 60de6a81aca).
1123:20231120:110604.440 ****** Enabled features ******
1123:20231120:110604.440 SNMP monitoring: YES
1123:20231120:110604.440 IPMI monitoring: YES
1123:20231120:110604.440 Web monitoring: YES
1123:20231120:110604.440 VMware monitoring: YES
1123:20231120:110604.440 SMTP authentication: YES
1123:20231120:110604.440 ODBC: YES
1123:20231120:110604.440 SSH support: YES
1123:20231120:110604.440 IPv6 support: YES
1123:20231120:110604.440 TLS support: YES
1123:20231120:110604.440 ******************************
1123:20231120:110604.440 using configuration file: /etc/zabbix/zabbix_server.conf
1123:20231120:110604.470 current database version (mandatory/optional): 06050143/06050143
1123:20231120:110604.470 required mandatory version: 06050143
1124:20231120:110604.490 starting HA manager
1124:20231120:110604.507 HA manager started in active mode
1123:20231120:110604.508 server #0 started [main process]
1126:20231120:110604.509 server #2 started [configuration syncer #1]
1125:20231120:110604.510 server #1 started [service manager #1]
1133:20231120:110604.841 server #9 started [lld worker #1]
1132:20231120:110604.841 server #8 started [lld manager #1]
1134:20231120:110604.841 server #10 started [lld worker #2]
If there was an error and the server was not able to connect to the database you would see something like this in the server log file :
10773:20231118:213248.570 Starting Zabbix Server. Zabbix 7.0.0alpha7 (revision 60de6a81aca).
10773:20231118:213248.570 ****** Enabled features ******
10773:20231118:213248.570 SNMP monitoring: YES
10773:20231118:213248.570 IPMI monitoring: YES
10773:20231118:213248.570 Web monitoring: YES
10773:20231118:213248.570 VMware monitoring: YES
10773:20231118:213248.570 SMTP authentication: YES
10773:20231118:213248.570 ODBC: YES
10773:20231118:213248.570 SSH support: YES
10773:20231118:213248.570 IPv6 support: YES
10773:20231118:213248.570 TLS support: YES
10773:20231118:213248.570 ******************************
10773:20231118:213248.570 using configuration file: /etc/zabbix/zabbix_server.conf
10773:20231118:213248.574 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to server on 'xxx.xxx.xxx.xxx' (115)
10773:20231118:213248.574 database is down: reconnecting in 10 seconds
10773:20231118:213258.579 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to server on 'xxx.xxx.xxx.xxx' (115)
10773:20231118:213258.579 database is down: reconnecting in 10 seconds
Let's check the Zabbix server service to see if it's enabled so that it survives a reboot
# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; preset: disabled)
Active: active (running) since Mon 2023-11-20 11:06:04 CET; 1h 2min ago
Main PID: 1123 (zabbix_server)
Tasks: 59 (limit: 12344)
Memory: 52.6M
CPU: 20.399s
CGroup: /system.slice/zabbix-server.service
├─1123 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
├─1124 "/usr/sbin/zabbix_server: ha manager"
├─1125 "/usr/sbin/zabbix_server: service manager #1 [processed 0 events, updated 0 event tags, deleted 0 problems, synced 0 service updates, idle 5.008686 sec during 5.016382 sec]"
├─1126 "/usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.092797 sec, idle 10 sec]"
├─1127 "/usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.027620 sec during 5.027828 sec]"
├─1128 "/usr/sbin/zabbix_server: alerter #1 started"
├─1129 "/usr/sbin/zabbix_server: alerter #2 started"
├─1130 "/usr/sbin/zabbix_server: alerter #3 started"
├─1131 "/usr/sbin/zabbix_server: preprocessing manager #1 [queued 1, processed 2 values, idle 5.490312 sec during 5.490555 sec]"
├─1132 "/usr/sbin/zabbix_server: lld manager #1 [processed 1 LLD rules, idle 5.028973sec during 5.029123 sec]"
├─1133 "/usr/sbin/zabbix_server: lld worker #1 [processed 1 LLD rules, idle 60.060180 sec during 60.085009 sec]"
├─1134 "/usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 60.065526 sec during 60.095165 sec]"
├─1135 "/usr/sbin/zabbix_server: housekeeper [deleted 0 hist/trends, 0 items/triggers, 0 events, 0 sessions, 0 alarms, 0 audit items, 0 autoreg_host, 0 records in 0.019108 sec, idle for 1 hour(s)]"
├─1136 "/usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.002856 sec, idle 59 sec]"
├─1137 "/usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000059 sec, idle 5 sec]"
├─1138 "/usr/sbin/zabbix_server: discovery manager #1 [processing 0 rules, 0.000000% of queue used, 0 unsaved checks]"
├─1139 "/usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000036 sec, idle 1 sec]"
├─1140 "/usr/sbin/zabbix_server: history syncer #2 [processed 1 values, 0 triggers in 0.005016 sec, idle 1 sec]"
├─1141 "/usr/sbin/zabbix_server: history syncer #3 [processed 0 values, 0 triggers in 0.000031 sec, idle 1 sec]"
├─1142 "/usr/sbin/zabbix_server: history syncer #4 [processed 0 values, 0 triggers in 0.000014 sec, idle 1 sec]"
├─1143 "/usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.005587 sec, idle 3 sec]"
├─1144 "/usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000010 sec, idle 5 sec]"
├─1145 "/usr/sbin/zabbix_server: self-monitoring [processed data in 0.000016 sec, idle 1 sec]"
├─1146 "/usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.002511 sec, idle 5 sec]"
├─1147 "/usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000009 sec, idle 1 sec]"
├─1148 "/usr/sbin/zabbix_server: poller #2 [got 1 values in 0.000232 sec, idle 1 sec]"
├─1149 "/usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000015 sec, idle 1 sec]"
├─1150 "/usr/sbin/zabbix_server: poller #4 [got 0 values in 0.000010 sec, idle 1 sec]"
This concludes our chapter on installing and configuring our Zabbix server. Next we have to configure our frontend. You can have a look at Installing Zabbix frontend with Nginx or Installing Zabbix frontend with Apache
Installing Zabbix frontend with Nginx¶
Before we can configure our frontend we need to install our package first. If you run the frontend on the same server as the Zabbix server then there is nothing else you have to do you can just run the following command on your server to install the packages needed for our frontend to install:
dnf install zabbix-nginx-conf and zabbix-web-mysql or if you used Postgres dnf install zabbix-web-pgsql
In case the frontend is on another server installed you need to add the Zabbix repository first like we did on our Zabbix server. In case you forgot or just skipped to this topic and don't know how to do this have a look at Adding the Zabbix repository
First thing we have to do is alter the Nginx configuration file so that we don't use the standard config.
vi /etc/nginx/nginx.conf
In this config look for the followin block that starts with :
server {
listen 80;
listen [::]:80;
server_name _;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
And place the following lines in comment:
server {
# listen 80;
# listen [::]:80;
# server_name _;
# root /usr/share/nginx/html;
We now have to alter the Zabbix configuration file so that it matches our setup. Edit the following file:
vi /etc/nginx/conf.d/zabbix.conf
server {
listen 8080;
server_name example.com;
root /usr/share/zabbix;
index index.php;
Replace the first 2 lines with the correct port and domain for your frontend in case you don't have a domain you can replace server_name with _; like in the exaple below:
server {
# listen 8080;
# server_name example.com;
listen 80;
server_name _;
root /usr/share/zabbix;
index index.php;
We are now ready to start our websever and enable it so that it comes online after a reboot.
systemctl enable php-fpm --now
systemctl enable nginx --now
Let's verify if the service is properly started and enabled so that it survives our reboot next time.
# systemctl status nginx
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/nginx.service.d
└─php-fpm.conf
Active: active (running) since Mon 2023-11-20 11:42:18 CET; 30min ago
Main PID: 1206 (nginx)
Tasks: 2 (limit: 12344)
Memory: 4.8M
CPU: 38ms
CGroup: /system.slice/nginx.service
├─1206 "nginx: master process /usr/sbin/nginx"
└─1207 "nginx: worker process"
Nov 20 11:42:18 zabbix-srv systemd[1]: Starting The nginx HTTP and reverse proxy server...
Nov 20 11:42:18 zabbix-srv nginx[1204]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Nov 20 11:42:18 zabbix-srv nginx[1204]: nginx: configuration file /etc/nginx/nginx.conf test is successful
Nov 20 11:42:18 zabbix-srv systemd[1]: Started The nginx HTTP and reverse proxy server.
The service is running and enabled so there is only 1 thing left to do before we can start the configuration in the GUI and that is to configure our firewall to allow incoming communication to the webserver.
firewall-cmd --add-service=http --permanent
firewall-cmd --reload
Open your browser and go to the url or ip of your frontend :
http://<ip or dns of the zabbix frontend server>/
If all goes well you should be greeted with a Zabbix welcome page. In case you have an error check the configuration again or have a look at the nginx log file :
/var/log/nginx/error.log
or run
journalctl -xe
This should help you in locating the errors you made.
When you point your browser to the correct URL you should be greeted with a page like here :
As you see there is only a limited list of local translations available on our Zabbix frontend to choose from
What if we want to install Chinese as language or another language from the list ? Run the next command to get a list of all locales available for your OS.
dnf list glibc-langpack-*
This will give you a list like
Installed Packages
glibc-langpack-en.x86_64
Available Packages
glibc-langpack-aa.x86_64
...
glibc-langpack-zu.x86_64
Let's search for our Chinese locale to see if it is available. As you can see the code starts with zh
# dnf list glibc-langpack-* | grep zh
glibc-langpack-zh.x86_64
glibc-langpack-lzh.x86_64
The command returns us 2 lines but as we have seen that the code was zh_CN we only have to install the first package.
# dnf install glibc-langpack-zh.x86_64 -y
When we return now to our frontend we are able to select the Chinese language.
Note
If your language is not available in the frontend don't panic it just means that there is no translation or that the translation was not 100% complete. Zabbis is free and relies on the community for it's translations so you can help in creating the translation. Go to the page https://translate.zabbix.com/
and help us to make Zabbix get better. Once the translation is complete the next Zabbix minor patch version should have your language included.
Click next when you are satisfied with the transaltions available. You will arrive at a screen to verifiy if all pre-requisites are met. If not fix them first but normaly it should be fine and you should be just able to click Next
The next page will show you a page with the connection parameters for our database.
First you select your DB type 'MySQL' or 'PostgreSQL' and fill in the IP or DNS name of the location of your database server. Use port 3306 for MariaDB/MySQL or 5432 if you used PostgreSQL.
Fill in the correct database name, in our case it was zabbix
.
If you used PostgreSQL then you also need to fill in the correct schema name in our case it was zabbix_server
Next line will ask you for the DB users here we created a user zabbix-web
. Enter it in the correct field and fill in the password that you used for this user.
Make sure the option Database TLS encryption
is not selected and press Next step
.
We are almost there. The only thing that rests us to do is give our instance a name, select our timezone and select a default time we like to use.
Press Next step
again you will see a page that tells you that the configuration is successful.
Press Finish to end the configuration.
We are now ready to login :
Login : Admin Password : zabbix
If you like to secure the frontend with SSL then checkout the following topic
Installing Zabbix frontend with Apache¶
Before we can configure our frontend we need to install our package first. If you run the frontend on the same server as the Zabbix server then there is nothing else you have to do you can just run the following command on your server to install the packages needed for our frontend to install:
dnf install zabbix-apache-conf and zabbix-web-mysql or if you used Postgres dnf install zabbix-web-pgsql
In case the frontend is on another server installed you need to add the Zabbix repository first like we did on our Zabbix server. In case you forgot or just skipped to this topic and don't know how to do this have a look at Adding the Zabbix repository
We are now ready to start our websever and enable it so that it comes online after a reboot.
systemctl enable php-fpm --now
systemctl enable httpd --now
Let's verify if the service is properly started and enabled so that it survives our reboot next time.
# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/httpd.service.d
└─php-fpm.conf
Active: active (running) since Mon 2024-03-04 08:50:17 CET; 7min ago
Docs: man:httpd.service(8)
Main PID: 690 (httpd)
Status: "Total requests: 96; Idle/Busy workers 100/0;Requests/sec: 0.213; Bytes served/sec: 560 B/sec"
Tasks: 278 (limit: 22719)
Memory: 39.6M
CPU: 1.132s
CGroup: /system.slice/httpd.service
├─ 690 /usr/sbin/httpd -DFOREGROUND
├─ 736 /usr/sbin/httpd -DFOREGROUND
├─ 737 /usr/sbin/httpd -DFOREGROUND
├─ 738 /usr/sbin/httpd -DFOREGROUND
├─ 739 /usr/sbin/httpd -DFOREGROUND
└─4534 /usr/sbin/httpd -DFOREGROUND
Mar 04 08:50:17 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
Mar 04 08:50:17 localhost.localdomain httpd[690]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set th>
Mar 04 08:50:17 localhost.localdomain httpd[690]: Server configured, listening on: port 80
Mar 04 08:50:17 localhost.localdomain systemd[1]: Started The Apache HTTP Server.x
The service is running and enabled so there is only 1 thing left to do before we can start the configuration in the GUI and that is to configure our firewall to allow incoming communication to the webserver.
firewall-cmd --add-service=http --permanent
firewall-cmd --reload
Open your browser and go to the url or ip of your frontend :
http://<ip or dns of the zabbix frontend server>/zabbix/
If all goes well you should be greeted with a Zabbix welcome page. In case you have an error check the configuration again or have a look at the Apache log file :
/var/log/httpd/error_log
or run
journalctl -xe
This should help you in locating the errors you made.
When you point your browser to the correct URL you should be greeted with a page like here :
As you see there is only a limited list of local translations available on our Zabbix frontend to choose from
What if we want to install Chinese as language or another language from the list ? Run the next command to get a list of all locales available for your OS.
dnf list glibc-langpack-*
This will give you a list like
Installed Packages
glibc-langpack-en.x86_64
Available Packages
glibc-langpack-aa.x86_64
...
glibc-langpack-zu.x86_64
Let's search for our Chinese locale to see if it is available. As you can see the code starts with zh
# dnf list glibc-langpack-* | grep zh
glibc-langpack-zh.x86_64
glibc-langpack-lzh.x86_64
The command returns us 2 lines but as we have seen that the code was zh_CN we only have to install the first package.
# dnf install glibc-langpack-zh.x86_64 -y
When we return now to our frontend we are able to select the Chinese language.
Note
If your language is not available in the frontend don't panic it just means that there is no translation or that the translation was not 100% complete. Zabbis is free and relies on the community for it's translations so you can help in creating the translation. Go to the page https://translate.zabbix.com/
and help us to make Zabbix get better. Once the translation is complete the next Zabbix minor patch version should have your language included.
Click next when you are satisfied with the transaltions available. You will arrive at a screen to verifiy if all pre-requisites are met. If not fix them first but normaly it should be fine and you should be just able to click Next
The next page will show you a page with the connection parameters for our database.
First you select your DB type 'MySQL' or 'PostgreSQL' and fill in the IP or DNS name of the location of your database server. Use port 3306 for MariaDB/MySQL or 5432 if you used PostgreSQL.
Fill in the correct database name, in our case it was zabbix
.
If you used PostgreSQL then you also need to fill in the correct schema name in our case it was zabbix_server
Next line will ask you for the DB users here we created a user zabbix-web
. Enter it in the correct field and fill in the password that you used for this user.
Make sure the option Database TLS encryption
is not selected and press Next step
.
We are almost there. The only thing that rests us to do is give our instance a name, select our timezone and select a default time we like to use.
Press Next step
again you will see a page that tells you that the configuration is successful.
Press Finish to end the configuration.
We are now ready to login :
Login : Admin Password : zabbix
In case you are like me and don't like the /zabbix path at the end of you url then there is an easy way to remove this. Edit you httpd
config file and add the lines below and replace it with your own domain:
vi /etc/httpd/conf/httpd.conf
NameVirtualHost 172.1.11.21:80
<VirtualHost "your ip or dns":80>
ServerName zabbixserver.mydomain.org
ServerAlias zabbixserver
DocumentRoot /usr/share/zabbix
</VirtualHost>
Don't forget to restart the httpd
service
systemctl restart httpd
Configure Zabbix HA¶
In this topic we will setup Zabbix in a High Available setup. This feature was added in Zabbix 6 and was one of the most important features added that time. The idea about this functionallity is that if your Zabbix server fails that another Zabbix server can take over. In this setup we will use 2 Zabbix servers but you are not limited to this you can add as many as you like.
The HA setup in Zabbix is rather basic but works like a charm so don't expect fancy things like load balancing.
Just like we did in our basic setup we will make a few notes again about the setup of the servers we have. I added the IP's that we will use here don't forgot to make notes of your own ip adresses.
Server | IP |
---|---|
Zabbix Server 1 | 192.168.0.130 |
Zabbix Server 2 | 192.168.0.131 |
Postgres DB | 192.168.0.132 |
Virtual IP | 192.168.0.135 |
Note
As you notice our DB is not HA this is not a Zabbix component you have to implement your own solution this can be a HA SAN or you DB in a HA cluster setup. The cluster setup of our DB is out of the scope and not related to Zabbix so we will not cover this here.
Let's install our Postgres DB¶
Note
If you are not running on x86 or like to try on another OS, then have a look at https://www.postgresql.org/download/ for the commands you need.
Warning
In this exercise we will take some shortcuts for the installation of the PostgreSQLDB and the OS. Look at our previous topics to get a better understanding where to tweak.
# Install the repository RPM:
sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
# Disable the built-in PostgreSQL module:
sudo dnf -qy module disable postgresql
# Install PostgreSQL:
sudo dnf install -y postgresql16-server
# Initialize the database and enable automatic start:
sudo /usr/pgsql-16/bin/postgresql-16-setup initdb
sudo systemctl enable postgresql-16 --now
Securing the PostgreSQL database¶¶
PostgreSQL works a bit different then MySQL or MariaDB and this applies aswell to how we manage access permissions.
Postgres works with a file with the name pg_hba.conf
where we have to tell who can access our database from where and what encryption is used for the password. So let's edit this file to allow our frontend and zabbix server to access the database.
# vi /var/lib/pgsql/16/data/pg_hba.conf
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host zabbix zabbix 192.168.0.130/32 scram-sha-256
host zabbix zabbix 192.168.0.131/32 scram-sha-256
host all all 127.0.0.1/32 scram-sha-256
After we changed the pg_hba file don't forget to restart postgres else the settings will not be applied. But before we restart let us also edit the file postgresql.conf and allow our database to listen on our network interface for incomming connections from the zabbix server. Postgresql will standard only allow connections from the socket.
# vi /var/lib/pgsql/16/data/postgresql.conf
Replace the line with listen_addresses so that PostgreSQL will listen on all interfaces and not only on our localhost.
listen_addresses = '*'
When done restart the PostgreSQL cluster and see if it comes back online in case of an error check the pg_hba.conf
file you just edited for typos.
# systemctl restart postgresql-16
For our Zabbix server we need to create tables in the database for this we need ot install the Zabbix repository like we did for our Zabbix server and install the Zabbix package containing all the database tables images icons, ....
Add the Zabbix repository and populate the DB¶
Add the Zabbix repo to your server (Don't forget to select the correct repo for your OS and Zabbix version) for this go to www.zabbix.com/download
# rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rhel/9/x86_64/zabbix-release-6.5-2.el9.noarch.rpm
Install the database scripts.¶
# dnf install zabbix-sql-scripts -y
Next we have to unzip the database schema files. Run as user root followin command::
# gzip -d /usr/share/zabbix-sql-scripts/postgresql/server.sql.gz
Create the DB users¶
Now we are ready to create our Zabbix users for the server and the frontend. If you like to separate users for frontend and server have a look at the basic installation guide.
# su - postgres
# createuser --pwprompt zabbix
Enter password for new role: <server-password>
Enter it again: <server-password>
Populate the database.¶
We are now ready to create our database zabbix. Become user postgres again and run next command to create the database as our user zabbix:
# su - postgres
# createdb -E Unicode -O zabbix zabbix
Let's upload the Zabbix SQL file we extracted earlier to populate our database with the needed schemas images users etc ... For this we need to connect to the DB as user zabbix.
# psql -U zabbix -W zabbix
Password:
psql (16.2)
Type "help" for help.
zabbix=> SELECT session_user, current_user;
session_user | current_user
--------------+--------------
zabbix | zabbix
(1 row)
zabbix=> \i /usr/share/zabbix-sql-scripts/postgresql/server.sql
CREATE TABLE
CREATE INDEX
CREATE TABLE
....
....
INSERT 0 1
DELETE 80424
COMMIT
Make sure the owner of your tables is the user zabbix;
zabbix=> \dt
List of relations
Schema | Name | Type | Owner
--------+----------------------------+-------+--------
public | acknowledges | table | zabbix
public | actions | table | zabbix
...
...
...
zabbix=> \q
Configure the firewall¶¶
One last thing we need to do is open the firewall and allow incoming connections for the PostgreSQL database from our Zabbix server because at the moment we dont accept any connections yet.
# firewall-cmd --new-zone=postgresql-access --permanent
success
# firewall-cmd --reload
success
# firewall-cmd --get-zones
block dmz drop external home internal nm-shared postgresql-access public trusted work
# firewall-cmd --zone=postgresql-access --add-source=<zabbix-serverip 1> --permanent
# firewall-cmd --zone=postgresql-access --add-source=<zabbix-serverip 1> --permanent
success
# firewall-cmd --zone=postgresql-access --add-port=5432/tcp --permanent
success
# firewall-cmd --reload
Now lets have a look to our firewall rules to see if they are what we expected:
# firewall-cmd --zone=postgresql-access --list-all
Our database server is ready now to accept connections from our Zabbix server :). You can continue with the next task
Install our Zabbix Cluster¶
Setting up a Zabbix cluster is not really different from setting up a regular Zabbix server obviously we need more then one. And there are also a few parameters that we need to configure.
Let's start by adding our Zabbix 7.0 repositories to our 2 Zabbix servers.
rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rhel/9/x86_64/zabbix-release-6.5-2.el9.noarch.rpm
Once this is done we can install our Zabbix servers on both systems.
dnf install zabbix-server-pgsql -y
We will now edit the config file on our first zabbix server. Run the next command:
vi /etc/zabbix/zabbix_server.conf
Once in the file edit the following lines to make our Zabbix server 1 connnect to the database
DBHost=<zabbix db ip>
DBName=zabbix
DBUser=zabbix
DBPassword=<your secret password>
In the same file we also have to edit another parameter to activate HA on this host.
HANodeName=zabbix1 (or whatever you like)
We are not done yet. We also have to tell Zabbix in case of a node fail to what server the frontend needs to connect.
NodeAddress=<Zabbix server 1 ip>>:10051
We are now done with the configuration of our 1st Zabbix server. Now let's do the same for our second server. I case you have more then 2 servers you can update them as well.
When you are done patching the config of your servers you can start the zabbix-server
service on both servers
systemctl enable zabbix-server --now
Let's have a look at the log files from both servers to see if it came online as we had hoped. on our first server we can run:
#grep HA /var/log/zabbix/zabbix_server.log
22597:20240309:155230.353 starting HA manager
22597:20240309:155230.362 HA manager started in active mode
Now do the same on our other node(s)
# grep HA /var/log/zabbix/zabbix_server.log
22304:20240309:155331.163 starting HA manager
22304:20240309:155331.174 HA manager started in standby mode
Installing the frontends¶
First things first before we can install and configure our webserver we need to install keepalived. Keepalived allows us to use a VIP for our frontends. Keepalived provides frameworks for both load balancing and high availability.
Info
Some useful documentation on the subject you might like. https://www.redhat.com/sysadmin/advanced-keepalived and https://keepalived.readthedocs.io/en/latest/introduction.html
Setup keepalived¶
So let's get started. On both our servers we have to install keepalived.
dnf install keepalived
We also need to adapt the configuration of keepalived on both servers. The configuration for both servers needs to be a bit changed so let's start with our server 1. Edit the config file with the following command:
# vi /etc/keepalived/keepalived.conf
Delete everything and replace it with the following lines:
vrrp_track_process track_nginx {
process nginx
weight 10
}
vrrp_instance VI_1 {
state MASTER
interface enp0s1
virtual_router_id 51
priority 244
advert_int 1
authentication {
auth_type PASS
auth_pass 12345
}
virtual_ipaddress {
192.168.0.135
}
track_process {
track_nginx
}
}
Replace enp0s1
with the interface name of your machine and replace the password with something secure. For the virual_ipaddress use aa free IP from your network.
Now do the same thing for our second Zabbix server.
# vi /etc/keepalived/keepalived.conf
Delete everything and replace it with the following lines:
vrrp_track_process track_nginx {
process nginx
weight 10
}
vrrp_instance VI_1 {
state BACKUP
interface enp0s1
virtual_router_id 51
priority 243
advert_int 1
authentication {
auth_type PASS
auth_pass 12345
}
virtual_ipaddress {
192.168.0.135
}
track_process {
track_nginx
}
}
Just as with our 1st Zabbix server, replace enp0s1
with the interface name of your machine and replace the password with something secure.
For the virual_ipaddress use aa free IP from your network.
Install and configure the frontend¶
On both servers we can run the following commands to install our webserver and the zabbix frontend packages:
dnf install nginx zabbix-web-pgsql zabbix-nginx-conf
Also let's not forget to configure our firewall
firewall-cmd --add-service=http --permanent
firewall-cmd --add-service=zabbix-server --permanent
firewall-cmd --reload
And now we can start our keepalived and enable it so that it comes up next reboot
systemctl enable keepalived nginx --now
Configure Zabbix Frontend¶
Click next till you see the following page and fill in the ip of your DB server. The port can be 0 this means we will use the default port. fill in the database name, user and password you used for the database. Make sure you deselect TLS encryption and select store passwords as plaintext. When you click next it won't work because we did not disable SELinux. Run the following command first on both Zabbix servers.
setsebool -P httpd_can_network_connect_db on
setsebool -P httpd_can_connect_zabbix on
This will allow your webservers to communicate with our database over the network. Now when we click next it should work.
We are almost ready the only thing left here is now to add the name of our server and configure the default timezone.
Since you’re using a host-based firewall, you need to add the necessary rules to permit IP protocol 112 traffic. Otherwise, Keepalived’s advertisement method won’t work.
firewall-cmd --add-rich-rule='rule protocol value="112" accept' --permanent
Now that this is all taken care of stop keepalived on our server and repeat the same steps on the second server. After this is finished start keepalived again.
Congratulations you have a HA Zabbix server now .
Checking the Database for HA info.¶
Now that everything is up and running there is probably something you like to know. Where can we find the info in our database ?
It's actually very straighforward we can go to our zabbix database and run the following query to see our servers: SELECT *FROM ha_node;
zabbix=# SELECT *FROM ha_node;
ha_nodeid | name | address | port | lastaccess | status | ha_sessionid
---------------------------+---------+---------------+-------+------------+--------+---------------------------
cltk7h2n600017kkd1jtx6f1f | zabbix2 | 192.168.0.131 | 10051 | 1710085786 | 0 | cltlov4ly0000jkkdteikeo77
cltk7ci340001inkc2befwg9f | zabbix1 | 192.168.0.130 | 10051 | 1710085787 | 3 | cltlov1r00000jtkcpeh9oqhp
This is also how our frontend is able to know what server it needs to connect to. Remember our picture in the first page ? Actually the frontend has a connection to our database and reads out the status from our zabbix server
.
This way it knows what server is active.
It's probably also good to know that we can have 4 statusses:
status | number | info |
---|---|---|
Active | 3 | Only one node can be active |
Standby | 0 | Multiple nodes can be in standby |
Stopped | 1 | A previous detected node is nog stopped |
Unavailable | 2 | A previous dtected node was lost without being shutdown |
Note
Zabbix agents need to have their Server and ServerActive addresses pointed to both active and passive Zabbix server. This option is supported in all agents since Zabbix 6.0
Ended: Getting started
The basics ↵
Zabbix Interface¶
This chapter is going to cover the basics we need to know when it comes to the Zabbix userinterface and the thing we need to know before we can start to fully dive into our monitoring tool. We will see how the userinteface works how to add a host, groups users, items ... so that we have a good understanding of the basics. This is something that is sometimes missed and can lead to frustrations not knowing why things don't work like we had expected them to work. So even if you are an advanced user it may be usefull to have a look into this chapter.
Overview of the interface¶
With Zabbix 7 the user interface after logging in is a bit changed. Our menu on the left side of the screen has has a small overhaul. Let's dive into it. When we login into our Zabbix setup the first time with our Admin user we see a page
like this where we have our main window
in green>
our main menu
marked in
red and our links
marked in yellow.
The main menu can be hidden by collapsing it completely or to reduce it to a set of small icons.
When we click on the button with the 2 arrows to the left:
You will see that the menu collapses to a set of small icons. Pressing ">>" will
bring the main menu
back to it's original state. Pressing the box with the arrow
sticking out next to the "<<" button will hide the main menu
completely.
To get the main menu back it's not too difficult we just look for the button on
the left with three horizontal lines and click it.
This will bring the menu back and clicking on the box with the arrow
again will bring the main menu
back.
Yet another way to make the screen bigger that is quit useful for monitors in NOK teams for example is the kiosk mode button. This one is however located on the left side of your screen and looks like 4 arrows pointing to every corner of the screen. Pressing this button will remove all the menus and leave only main window to focus on.
When we want to leave the kios mode, the button will be changed to 2 arrows pointing to the inside of the screen. Pressing this button will revert us back to the original state.
Tip
We can also enter and exit kiosk mode by making use of parameters in our Zabbix url:
/zabbix.php?action=dashboard.view&kiosk=1
- activate kiosk mode
or /zabbix.php?action=dashboard.view&kiosk=0
- activate normal mode
Note
There are many other page parameters we can use. A full list can be found here Zabbix also has a global search menu that we can use to find hosts, host groups and templates.
If we look in the search box for server
you will see that we get an overview
of all templates, host groups and hosts with the name server
in it.
Main menu¶
Our main menu on the left consists of a few sections, 9 to be exact:
Menu Name | Details |
---|---|
Dashboards | Contains an overview of all the dashboards we have access to. |
Monitoring | Shows us the hosts, problems, latest data, maps, ... |
Services | An overview of all the Services and SLA settings. |
Inventory | An overview of our collected inventory data. |
Reports | Shows us the system information, scheduled reports, audit logs, action logs, etc . |
Data collection | Contains all things related to collecting data like hosts, templates, maintenance, discovery, ... |
Alert | The configuration of our media types, scripts and actions |
Users | User configuration like user roles, user groups, authentication, API tokes, ... |
Administration | The administration part containing all global settings, housekeeper, proxies, queue, ... |
Links menu¶
Our last part the links
part contain a set of useful links that we can use:
Menu name | Details |
---|---|
Support | This brings us to the technical support page that you can buy from Zabbix. Remember that your local partner is also able to sell these contracts and can help you in your own language. Your local distributors |
Integrations | The official zabbix integeration page |
Help | The link to the documentation of your Zabbix version |
User settings | The user profile settings. |
Sign out | Log out of the current session. |
There are still a few buttons that we need to cover on the right side of our screen
The edit button allows us to change our dashboard. This is something we will cover later. On the far left side there is a "?" this will bring you to the Zabbix documentation page that explains everything about the dashboard. The button on the right side with the 3 horizontal lines is the one to share, rename, delete, ... our dashboards.
System Information¶
There is also a box on the dashboard called System Information
. This widget
will show you the current System status of your Zabbix setup. Let's go over
the different lines of information as they are important to understand.
Parameter | Value | Details |
---|---|---|
Zabbix server is running | This gives us the status of our zabbix server if it is running yes or no and if it is running on our localhost or another IP and on what port the zabbix server is listening. If no trapper is listening the rest of the information can not be displayed | IP and port of the Zabbix server |
Zabbix server version | This shows us the version of the Zabbix server so the version you see at the bottom of your screen is the one from the Zabbix frontend and can be different but should be in the same major version. |
Version Number |
Zabbix frontend version | This is the version of the frontend and should match with what you see at the bottom of your screen. | Version Number |
Number of hosts (enabled/disabled) | The total number of hosts configured on our system | How many of those are enabled and disabled |
Number of templates | The number of templates installed on our Zabbix server. | |
Number of items (enabled/disabled/not supported) | This line shows us the number of items we have configured in total in this case 99 | 90 are enabled and 0 are disabled but 9 of them are unsupported. This last number is important as those are items not working. We will look into this later why it happens and how to fix it. For now remember that a high number of unsupported items is not a good idea. |
Number of triggers (Enabled/disabled[problem/ok]) | The number of triggers configured | Number of enabled and disabled triggers. Just as with items we also see if there are triggers that are in a problem state or ok state. A trigger in a problem state is a non working trigger something we need to monitor and fix. We will cover this also later. |
Number of users (online) | Here we see the number of users that are configured on our system | The nunber of users currently online. |
Required server performance, nvps | The number of new values per second that Zabbix will process per second. | This is just an estimated number as some values we get are unknown so the real value is probably higher. So we can have some indication about how many IOPS we need and how busy our database is. A better indication is probably the internal item zabbix[wcache,values,all] |
High availability cluser | It will show us if we are running on a Zabbix HA cluster or not | Failover delay once HA is activated |
Tip
System information may display some additonal warnings like when your database doesn't have the correct character set or collation UTF-8. Also when the database you used is lower or higher then the recommended version or when there are misconfigurations on housekeeper or TimescaleDB. Another warning you can see is about database history tables that aren't upgraded or primary keys that have not been set. This is possible if you are coming from an older version before Zabbix 6 and never did the upgrade.
The main menu explained¶
It's important to know that we have so far seen our dashboard with the Admin user
and that this user is a Zabbix Super Admin
user. This means that the user has
no restrictions. Zabbix works with 3 different levels of users we have the regular
users, Zabbix Admin and Zabbix Super Admin users. Let's have a look
- A
Zabbix User
will only see the red part of ourmain menu
and will only be able to see our collected data. - A
Zabbix Admin
will see the red part and the yellow part of themain menu
and is able to change our configuration. - A
Zabbix Super Admin
will see the completemain menu
and so is able to change the configuration and all the global settings.
- Problems: This page will give us an overview of all the problems. With filter we can look at recent problems past problems and problems that are active now. There are many more filters tor drill down more.
- Hosts: This will give us a quick overview page with whats happening on our hosts and allows us to quickly go to the latest data, graphs and dashboards.
- Latest data: This page I probably use the most, it shows us all the information collected from all our hosts.
- Maps: The location where we can create map that are an oveview of our IT infrastructure very useful to get a high level overview of the network.
- Discovery: When we run a network discovery this is the place where we can find the results.
- Services This page will give us a high level overview of all services configured in Zabbix.
- SLA: An overview of all the SLAs configured in Zabbix.
- SLA Report: Here we can watch all SLA reports based on our filters.
- Overview: A place where we can watch all our iventory data that we have retrieved from our hosts.
- Hosts: Here we can filter by host and watch all inventory data for the hosts we have selected.
- System information: System information is a summary of key Zabbix server and system data.
- Scheduled reports: The place where we can schedule our reports, a pdf of the dashboard that will be sent at a specified time and date.
- Availability report: A nice overview where we can see what trigger has been in ok/nok state for how much % of the time
- Top 100 triggers: Another page I visit a lot here we have our top list with triggers that have been in a nok state.
- Audit log: An overview of the user activity that happend on our system. Useful if we want to know who did what and when.
- Action log: A detailed overview of our actions can be found here. What mail was sent to who and when ...?
- Notifications: A quick overview of the number of notifications sent to each user.
- Template groups: A place to logical group all templates together in different groups. Before it was mixed together with hosts in host groups.
- Host groups: A logical collection of different hosts put together. Host groups are used for our permissions.
- Templates: A set off entities like items and triggers can be grouped together on a template, A template can be applied to one or more hosts.
- Hosts: What we need in Zabbix to monitor A host, application, service ...
- Maintenance: The place to configure our maintenance windows. A maintenance can be planned in this location.
- Event correlation: When we have multiple events that fires triggers related we can configure correlations in this place.
- Discovery: Sometimes we like to use Zabbix to discover devices, services,... on our network. This can be done here.
- Actions: This menu allows us to configure actions based on
events
in Zabbix. We can create such actions for triggers, services, discovery, autoregistration and internal events. - Media types: Zabbix can sent messages, emails etc ... based on the actions we have configured. Those media types need templates and need to be activated.
- Scripts: In Zabbix it's possible to make use of scripts in our actions and frontend. Those actions need to be created here first and configured.
- User groups:
- User roles:
- Users:
- API tokens:
- Authentication:
- General:
- Audit log:
- Housekeeping:
- Proxies:
- Macros:
- Queue:
Info
More information can be found in the online Zabbix documentation here
Info
You will see that Zabbix is using the modal forms in the frontend on many places.
The problem is that they are not movable. This
module created by one of the Zabbix devs UI Twix
will solve this problem
for you.
Note
At time of writing there is no Dashboard import/export functionality in zabbix. So when upgrading dashboards it needs to be created by hand. It was on the roadmap for 7 but didnt made it so feel free to vote https://support.zabbix.com/browse/ZBXNEXT-5419
Zabbix Users & User groups¶
Now that we know how the Zabbix dashboard is build up our first task will be to create a user. In case you missed it the standard Zabbix (yes the capital Z here is eeded to login.) user is Admin and has the password zabbix so we need to change this ASAP.
The most confusing part is probably that the user Admin in zabbix is actually a super admin
but more about that later.
In this topic we will cover:
- Changing the Zabbix super admin password
- Zabbix User types
- Creating a new User in Zabbix
- Strengthen the Zabbix password policy.
- User Media
- User permissions
- User Roles
- User Groups
- User Groups Overview
Changing the Zabbix super admin password¶
In our menu on the right side of the screen, click the Users
section, and then choose users
. As you can see here in the screenshot.
You will now see a list of all the users that are created on the system when installing a new Zabbix instance. Here you will always see a list of all users that are configured on the system.
To change the password, do the following steps:
- Click user Admin
- Click on the button Change password
.
- Fill in the current password, zabbix
- Fill in the new password twice and press Update
at the bottom of the page.
Zabbix User types¶
Before we create new users, it's important to know that Zabbix has three user types that are built-in.
User type | Description |
---|---|
Zabbix User | This is a normal user that only has read-only permissions if given. So there are no permissions assigned by default. |
Zabbix Admin | A user with read/write permissions. Just like the Zabbix user, there are no permissions by default. However access can be denied to some groups. |
Zabbix Super Admin | A user with group read/write permissions. The user will have read/write access to all host and template groups. Access can't be revoked by denying access to groups, like with a normal admin. |
Besides these differences, these users also have different access rights to our menu. Let's have a closer look.
- A normal user will only see a part of our menu on the left. Also, some sub-sections of the menu will not be visible. An
Admin
user will have more rights than a regular user and will be able to make some configuration changes in Zabbix. ASuper Admin
user will have unlimted right and see every part of the menu. The only way to limit aSuper Admin
will be by making use of roles. Something we cover later. - An
Admin
user will have more rights than a regular user and will be able to make some configuration changes in Zabbix. - A
Super Admin
can access all parts of the menu.
This table gives an overview of all the permissions a Zabbix user, admin, and super admin have in the Zabbix menu:
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Dashboards | ✅ | ✅ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Monitoring | ✅ | ✅ | ✅ |
- Problems | ✅ | ✅ | ✅ |
- Hosts | ✅ | ✅ | ✅ |
- Latest data | ✅ | ✅ | ✅ |
- Maps | ✅ | ✅ | ✅ |
- Discovery | ❌ | ✅ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Services | ✅ | ✅ | ✅ |
- SLA | ❌ | ✅ | ✅ |
- SLA Reports | ✅ | ✅ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Inventory | ✅ | ✅ | ✅ |
- Overview | ✅ | ✅ | ✅ |
- Hosts | ✅ | ✅ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Reports | ✅ | ✅ | ✅ |
- System information | ❌ | ❌ | ✅ |
- Scheduled reports | ❌ | ✅ | ✅ |
- Availability report | ✅ | ✅ | ✅ |
- Triggers top 100 | ✅ | ✅ | ✅ |
- Audit log | ❌ | ❌ | ✅ |
- Action log | ❌ | ❌ | ✅ |
- Notifications | ❌ | ✅ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Data Collection | ❌ | ✅ | ✅ |
- Template groups | ❌ | ✅ | ✅ |
- Host groups | ❌ | ✅ | ✅ |
- Templates | ❌ | ✅ | ✅ |
- Hosts | ❌ | ✅ | ✅ |
- Maintenance | ❌ | ✅ | ✅ |
- Event correlation | ❌ | ❌ | ✅ |
- Discovery | ❌ | ✅ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Alerts | ❌ | ✅ | ✅ |
- Trigger actions | ❌ | ✅ | ✅ |
- Service actions | ❌ | ✅ | ✅ |
- Autoregistration actions | ❌ | ✅ | ✅ |
- Internal actions | ❌ | ✅ | ✅ |
- Media types | ❌ | ❌ | ✅ |
- Scripts | ❌ | ❌ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Users | ❌ | ❌ | ✅ |
- User groups | ❌ | ❌ | ✅ |
- User roles | ❌ | ❌ | ✅ |
- Users | ❌ | ❌ | ✅ |
- Api tokens | ❌ | ❌ | ✅ |
- Authentication | ❌ | ❌ | ✅ |
Zabbix User | Zabbix Admin | Zabbix Super Admin | |
---|---|---|---|
Administration | ❌ | ❌ | ✅ |
- General | ❌ | ❌ | ✅ |
- Audit log | ❌ | ❌ | ✅ |
- Housekeeping | ❌ | ❌ | ✅ |
- Proxies | ❌ | ❌ | ✅ |
- Macros | ❌ | ❌ | ✅ |
- Queue | ❌ | ❌ | ✅ |
- An
Admin
user will have more rights than a regular user and will be able to make some configuration changes in Zabbix. - A
Super Admin
can access all parts of the menu.
Creating a new User in Zabbix¶
So now that we are in the users section of Zabbix, it's probably a good time to create a new user for our system. If you skipped the previous step, go to the menu Users
-> Users
.
Click on the top right on Create user
and fill in the details of your new users. You will see that some fields have red asterisks in front of them, like Username and Password, ... this means that those fields are mandatory to fill in.
Strengthen the Zabbix password policy.¶
Zabbix passwords rely on a minimum length of 8 characters and also block a list of easy-to-guess passwords. We can make our passwords more secure by telling Zabbix that our passwords must contain uppercase and lowercase characters, a digit, and a special character. This policy is a global policy that will be enforced, and we have to set this policy as Super Admin. Go to the menu Users -> Authentication. In older versions, you can find it under Administration Authentication.
Parameter | Description |
---|---|
Username | A unique name that will be used as username when we login. |
Name | The users firstname this field is optional visible in acknowledgment information and notification recipient information if set. |
Last Name | Users last name. Optional, this field is optional visible in acknowledgment information and notification recipient information if set. |
Groups | Select what group the user will belong to. Atleast 1 group needs to be selected. This feeld will auto complete or you can press the '''Select''' button at the end of the field |
Password | There are 2 password fields they can only be used for internal authentication but more about this later. If the user has the Super admin role then clicking on the Change password button opens an additional field to entering the current (old) password. On a successful password change, the user for which the password was changed will be logged out of all active sessions. |
Language | Language of the frontend. The php gettext extension is required for the translations to work. And the language needs to be configured on the system. See the chapter "Installing Zabbix" in case you forgot. |
Timezone | Select the time zone per user or use the default timezone that is configured on the Zabbix server. |
Theme | Here users can select their own look and feel by choosing one of the 4 themes provided by Zabbix or another custom made theme. Default will switch to the default theme chosen by the admin. |
Auto-Login | Check this box so that the user will be remembered for 30 days. The browser must accept cookies for this to work. |
Auto-Logout | Checking this box makes sure the user gets logged out automatically, after the set amount of seconds (minimum 90 seconds, maximum 1 day). Time suffixes are supported, e.g. 90s, 5m, 2h, 1d. Note that this option will not work if : |
Refresh | Set the refresh rate used for graphs, plain text data, etc. Can be set to 0 to disable. Time suffixes are supported. Ex: 90s, 5m, 1h. |
Rows per page | Define how many rows per page will be displayed in lists. |
URL(after login) | You can make Zabbix transfer the user to a specific URL after successful login. This can be useful for monitors in NOC team for example so you arrive on a specific dashboard that is maximised. You can make Zabbix transfer the user to a specific URL after successful login. This can be useful for monitors in NOC team for example so you arrive on a specific dashboard that is maximised. |
User Media¶
The tab ''' Media ''' contains a list of all media that are defined for our user. Media is used for sending notifications to the user. We can click the Add
button.
Adding the media here is not enough to receive notification; we also need to configure our media properly, and we still need to configure actions as well. When pressing the ''' Add ''' button, we get a popup where we can select some information.
Parameter | Description |
---|---|
Type | A drop down list with the names of all media types. When a media type is disabled it will be in red. |
Send to | Here we can provide contact information. For an email media type it is possible to add several addresses by clicking on '''Add''' below the address field. In this case, the notification will be sent to all email addresses provided. It's also possible to specify recipient name in the Send to field of the email recipient in a format 'Recipient name address1@company.com'. Note that if a recipient name is provided, an email address should be wrapped in angle brackets (<>). UTF-8 characters in the name are supported, quoted pairs and comments are not. For example: John Doe manager@open-future.com and manager@nycdatacenter.com are both valid formats. Incorrect examples: John Doe manager@open-future.com, %%"Zabbix\@\<H(comment)Q>" zabbix@company.com %%. |
when active | The time when media will be active from monday till sundat, 1-7 and the time from 00:00 till 24:00 for example only in weekends from 6 in the morning till 5 in the evening: 6-7,06-17:00i. This is based on the user his timezone |
Use if severity | A list of checkboxes from the severities you would like to recieve notifications from. Selected severities will be displayed in color. !! Read the warning below!! |
Status | Status of the media we have selected either enabled or disabled ( in use or not ) |
Warning
When selecting the different severity levels, be aware that you have to select Not classified
if you want to receive notifications about non-trigger events, like internal events. For more information, check out Event Sources. This is something that is not obvious, and Zabbix documentation could be better at explaining this.
User permissions¶
When we go to the Permissions
tab in our Users
, we will get an overview of all permissions our users had in the menu structure. Or when creating a new user, we have the option to select a User Role
. Zabbix has four different User Roles
built-in. There is a User role, Admin role, Super admin role, and a Guest role.
The Guest role
is a role with very strict access limitations. Its role is intended for users to access Zabbix without any user account. I never advise using this role unless you know what you are doing. When you open your GUI to users without any authorization, this could leak potential sensitive data like hostnames, IPs, etc.
Choosing a User type
is one thing; based on the User type
we choose, our users will have more or less rights in our main menu. But there is another important part when choosing the User Type
. This also has an impact on the rights each user has over host groups. For example, a regular user can only have read rights or no rights. A Zabbix admin user can have full, read-only, or no rights, and a Zabbix Super Admin always has full rights on host groups, and his rights on the host groups cannot be revoked.
Here is an overview of every user and his rights:
Group rights | Zabbix User | Zabbix Admin | Zabbix Super Admin |
---|---|---|---|
Read/Write | Read Only | Full | Full |
Read Only | Read Only | Read Only | Full |
Deny | None | None | Full |
Note
With all this knowledge, we now know that if we want to create a regular user who also has access to certain parts of the Administration menu, that it's not possible. We can never create a user that has only RO access to certain host groups and RW access to the Administration
part. What we could do, however, is create a Super Administrator
account and remove access from the menu for certain parts of the Administration
menu by creating a special role. There is no limit on the number of roles you can create.
Note
Also, be aware that when you click on an item on the dashboard on Update
, you will see a modal window popup with some options to change the severity, close a problem, etc., so some will be greyed out. This is because the user needs write permissions. For example, a user needs write permissions to close a problem and change the severity level.
Note
With Zabbix 7 Permission checks have been made much faster. This was made possible by making some improvements on how permissions are stored. This should make the frontend faster when when we have permission havy pages to load like the ones with hosts or problems widgets.
- New tables have been introduced for the check of non-privileged users.
- The new tables will keep hashes (SHA-256) of user group sets and host group sets for each user/host.
- Also a new permission table was introduced for storing only the accessible combinations of users and hosts, specified by the hash IDs.
- Hashes and permissons are not calculated for Super Admin users.
User Roles¶
User roles have been in Zabbix since version 5.2 and make our lives easier by allowing us to make some custom adjustments to the standard defind user types in Zabbix.
When we go to our Permissions
tab, we can see a box Role
. Press the Select
box to see a popup with a list of roles to choose from. There are four standard roles to choose from.
You can create your own list of rules by going to the menu Users -> User Roles
and create your own limited user.
The box is marked with an asterisk in front, so you need to select a user role for every user you create.
Warning
Be aware that no permissions can be added to user roles only permissions can be revoked.
User Groups¶
A user always needs to be member of one or more User groups
. We will not set any user rights directly on Users in Zabbix but we do this on User groups. So if a User needs the permission to view or edit a host or a template then this is set on the User group
wich has the permisson to view or edit a host or template group and never on a host or a template directly.
Zabbix has a few different rights we can use on group level, as we have seen above. To make it easier for you I add them again:
Group rights | Zabbix User | Zabbix Admin | Zabbix Super Admin |
---|---|---|---|
Read/Write | Read Only | Read/Write | Read/Write |
Read only | Read Only | Read Only | Read/Write |
Deny | None | None | Read/Write |
- Read Only : The user has only read access and cannot modify anything.
- Read/Write : The user can see and edit.
- Deny: There is not even the options to see any data
When it comes to permissions in Zabbix groups, the highest level will win.
A user that has read and read-write rights on the same host will get read-write permissions. Except for Deny, Deny will always overrule.
So if we have a Zabbix Admin
user then this user can have Read/Write rights, if we add a host in a hostgroup where our usergroup has read rigths, and the same server is in another hostgroup with Read/Write rights, then our user will have Read/Write permissions on the hosts.
However if the same host is only in the Read
hostgroup then our user will only have read rights.
If we also add host in a Hostgroup where our usergroup has Deny
rights then the server will not be visible.
Let's have a look at our User groups
, for this go to the menu Users -> User groups
and click on one of the existing users. I used Guest
in this case.
User Groups Overview¶
Under the tab User group
we see the following options:
- User group : A field where we have to specify a unique name. This field in mandatory
- Users : Here we add users to our group. Users need to exist before we can add them. Just press select or type the name.
-
Frontend Access : How users of the group will authenticate with Zabbix.
- System default : The global configure access method
- Internal : The most easiest way User and Password are configured in Zabbix ( Ignored if HTTP authentication is the global default.)
- LDAP: LDAP/AD authentication ( Ignored if HTTP authentication is the global default )
- Enabled : If checked the group is Enabled else it will be Disabled
- Debug : Activate debug mode for the users in this group More info about debug
-
The next tab next to
User group
is the tabTemplate permissions
. Here we can define whatUser group
will have access to whattemplate group
. We can define if aUser group
has read, read-write permissions or if all access must be denied. When selecting a template group don't forget to press theAdd
button first so that you see theTemplate group
appear in the Permissions box. Then when you are ready confirm again at the bottom of the page withUpdqte
.
- The
Hosts permissions
tab allows us to specify whatUser group```` will have what kind of access on the selected
Host groupsthis can again be read, read-write or explicit deny. Just as with the
Templates permissionstab don't forget to click
Addfirst and when you are ready defining all the permissions click
Update``` at the bottom. The name is a bit confusing as we don't select permissions for a host but a host group.
Note
If we add multiple lines with the same host group or template group with different permissions Zabbix will apply the strongest permission.
Alow be aware that a Super admin user can enforce nested groups to have the same level of permissions as the parent group. It can be done in the host group or template group configuration.
- The
Problem tag filter
allows us to filter problems based on tags and their value. It also allows us to separate the access to host groups from our possibility to see only the problems we want.
Let's do this together:¶
Let us make three Host groups
, go to the Data collection
menu -> Host groups
and create a Host group for read
, read-write
, and deny
.
Next step is to create a host and add the host in our three groups. Go to the Data collection
menu -> Hosts
and press Create host
on the right. Add a Host name
, the name is not that important and add the three Host groups
we just made.
The only thing we need to do now is create our User
and User group
and give the correct rights. Go to our menu Users
-> Users group
and click on the top right to Create user group
. Let's call this group our Admin Group
as we need a Zabbix Admin
that we can give read, read-write and later deny to show this.
Next go to the tab Host permissions
and start typing the name of our group read
in the search box or press the Select
button and select the correct group. Next before we do anything select also the correct permissions Deny
and press the add just below NOT the button.
Do this also for the group read-write
and deny
. If everything looks like in our screenshot then press the Add
button
Now for the final step let's create a user. Go to the menu Users
-> Users
and create a new user, in the field Username
we can add our fictive user with the name Brian.
In the Groups
box we select our Users group
this was Admin Group
. Don't forget also to add a Password we need to do this twice.
Next go to the tab Permissions
and select the role Admin role
.
You will see directly once selected that our users bridan has read, write and deny on the correct groups. Press Add
at the bottom.
Now it's time to check if everything is as expected. Our user Brian
if all goes well shouldn't have any rights as we explicitly denied accesss.
Press Sign out
at the bottom left and then login as user Brian
.
Go to the menu Monitoring
-> Hosts
. Select all the hosts groups, you should normally only see read, and read-write. Our host group Deny
is not visible and our host postgres
is not visible either.
Now log back in as user Admin
, our Zabbix Super Admin and remove the deny group from our Admin group
. This can be done by selecting the None
permissions for the group Deny
in the Host permissions
tab from our User group
.
Log back in as our user Brian go back to the Monitoring
menu to Hosts
. If all goes well our groups read
and read-write
are still selected if nog you just select them again. You will see that our host postgres
is visisble and that you can click on it to edit the host propreties.
As final test you can try to remove the group read-write
same as we did before with the Deny
group. This time only the read
group will be visible for our user and Brian will not be able to edit our host postgres
anymore.
Let's try out tags¶
Now let's add tags into the mix. Imagine that we only like to see problems with a tag read-write
and value off
.
Go to User groups
select our Admin Group
again and go to the tab Problem tag filter
and fill in the needed tag read-write
and value off
.
Now we need to create a problem for this we will add an item and a trigger to our host postgres
. Go to the menu Data collection
-> Hosts
and click on items behind our host postgres
. On the top right you will see a button Create item
click on it and fill in the same data as in the screenshot below. Don't worry if you don't understand anything we will come to items later.
Note
In this item we just tell our Zabbix server to do a ping to IP 192.168.10.1
make sure this IP doesn't exist in your lan so try to ping it first to be sure you don't get a reply back. If you do get a reply back change the IP with some address that is not pingable for you.
Next step once you have filled in all the data is to save the item and click on top on Triggers. You will also notice now that there is a 1
next to Items. This indicates that we have made 1 item on our host postgres
.
Now that we are in the trigger tab click in the top right corner on the button Create trigger
.
Once again copy over all the data from the screenshot and save the trigger. If you changed the IP in the item make sure you use same IP in the trigger.
Next let's add a tag on our host postgres
that tells Zabbix to mark everything on the host with a tag read-write
and value on
. Remember we added a value off
in our User group
problem tag filter tab. So we only want to see everything with a tag read-write
and value off
.
When you go now to the Problem
page in the menu Monitoring
you should see after some time a warning that there is a problem on our host postgres. You will also see that the problem got a tag read-write with value on
.
You can clearly see that under our Zabbix super admin
user the problem is visible. Now do the same but as user Brian
. You will notice that there is no visible problem for our user even he has read-write
access to the hostgroup where our server postgres
belongs to.
Now as user Brian
I would like to see the problem so let's go to our menu Data collection
and click on our host postgres
. Go to the Tags
tab and change the value from our tag read-write
from on to off
. So now everything on our host should get the tags read-write
with value off
. So now Brian should be able to see the problem right ?
Sadly Brian
is still not able to see the problem in our Problem page. This is because the problem was already created in Zabbix and has already received the tag. So the only way to fix this is to close the problem first and let Zabbix create a new problem again.
As Super Admin
log back in and go to our trigger Ping
and mark the box Allow manual close
and press Update
.
Go back to the dashboard and behind the problem ping you will see Update
. Click on it and selec the option Close problem
and press Update
.
Log back in as our user Brian
and go to the problem dashboard. We will see that the problem is back. Even we closed the problem before Zabbix opened a new problem because the issue was not resolved. This time our issue has the tag with the correct value.
Note
A Zabbix user needs to be created with a user role. You cannot create one without.
Warning
Be careful if you use the API at the time of writing it's possible to create a Zabbix user with the API without a role. When created by the API the user can even be saved by the frontend afterwards !
Info
More information can be found in the online Zabbix documentation here
Zabbix hosts¶
To understand how Zabbix works, it's important to know that Hosts
in Zabbix are a
reference to anything we would like to monitor. It can be a physical host,
a virtual machine, an application, a device, or even just a dummy host used
to calculate data from existing hosts into something new.
It's probably one of the first tasks that we will do as an Admin when we first login to Zabbix because we need a host if we would like to monitor some metrics. It's however, important to know that hosts cannot be created without being in a hostgroup.
In this chapter we will teach you:
- How to add a host in Zabbix
- Host configuration tabs
- IPMI
- Tags
- Macros
- Inventory
- Encryption
How to add a host in Zabbix¶
With this said, let's see how to create our first host.
Let's go to the menu on your left
and select Data Collection
-> Hosts
.
We see that there is already a host configured and that the availability icon
is "RED". Don't worry about it, this is normal. We have no Zabbix agent
installed or configured.
To add a new host to our system, we have to press Create host
,
this button can be found in the upper right corner of our screen.
We now get a modal form where we need to fill in some information about our host. The fields marked with a red asterisk "*" are the fields that are mandatory.
Parameter | Description |
---|---|
Host name | Here we need to enter the Host name of the machine we would like to add. The name can contain alphanumerics, spaces, dots, dashes, and underscores. HOWEVER you are not allowed to use leading and trailing spaces. The Host name in the frontend is what we need later for the configuration of our Zabbix agent, so make sure you remember it. |
Visible name | The host name, as we have seen, is needed to configure our Zabbix agent. So in case you like to give it a unique name or one that is randomly generated, ... you can add a visible name here. This name will then be used on the frontend instead of what we call the technical name host name. This name has support for UTF-8, so special characters are supported. This name will be used in all the places like maps, the latest data, inventory, ... |
Templates | Templates are like blueprints that we can use on our hosts to add items, triggers, etc. We explain more about it in the topic Zabbix templates. You can start typing the name of the template, and Zabbix will start to show a list with matches, or you can press the Select box and choose one from the list. |
Host groups | Every host must belong to atleast one host group . This is because permissions are set on host groups. You can type the name of the host group, and a list of matching groups will start to appear. Another way is to select a host group from an existing list by pressing the Select button. Or you can create a new group by just typing the name and pressing on the box that shows the name of the group you typed with (new) behind it |
Interfaces | Zabbix supports several host interfaces, like the Zabbix agent, SNMP, JMX, and IPMI. By default, when we create a host, no interface is added. To add an interface, press Add and fill in the needed information, like IP or DNS, depending on the host interface chosen. When an interface is in use (items created that use the interface), then the interface cannot be removed. |
Description | A place to enter a short description about our host. |
Monitored by proxy | If we have proxies configured, we can select them here if we like to monitor our host through a proxy. |
Enabled | Mark the checkbox to enable the host. This will keep it monitored by Zabbix. When unchecked, the host will not be monitored. |
Host configuration tabs¶
Before we add a host ourselves, there are a few things we need to know first.
When we click on a host that we have already configured, there are a few things
that we will notice. First of all, we see a blue
line under Host
.
This means that we are on the current tab
of the host page.
As you can see, there are multiple tabs that we can click on, like IPMI, Tags, Macros,...
The next thing we see is that next to the tab Macros
, there is a number 2.
This is because there are two macros configured in the macro tab.
So when we add information to tabs like macros or tags ... ,
Zabbix will show how many items we have added to these tabs by showing next
to the tab name the number.
When looking at the encryption
tab, we notice the green dot.
This shows us that an option on the tab has been activated.
Now that we know this, let's get a quick overview of every tab and see what it does.
IPMI¶
So looking at the IPMI
tab, there are a few things we need to fill in when
working with an IPMI interface. IPMI stands for Intelligent Platform Management
Interface and is basically a set of standards to manage hardware platforms.
In short, it allows us to monitor and manage our servers hardware even if
the server is not turned on yet. IPMI is better known as ILO on HP servers
and DRAC on Dell servers.
Parameter | Description |
---|---|
Autentication algorithm | Select the authentication algorithm that we have configured on our IPMI server this can be Default, none, MD2, MD5, Straight, OEM, RMCP+ |
Privelege level | Here we select the privelege level: Callback, User, Operator, Admin or OEM. |
Username | The user for authentication that was created on the host. User Macros can be used. |
Password | The password for our user on the host. User macros can be used. |
Note
We will cover IPMI in more detail later in the Chapter IPMI Monitoring
Tags¶
The Tags tab allows you to define host-level tags. Host-level tags are used to provide additional information about the host that can be used as a filter in the webUi.
example:
We add a datacenter
tag to physical hosts servers to indicate the location of
the server and thus make it easy to list all the servers present in this datacenter,
impacted by incident.
Macros¶
The macros tab allows you to define host-level user macros. Macros in Zabbix can be seen as variables that can be overloaded at will. You need to use and abuse them to make templates as reusable as possible.
Note
We will cover Macro in more detail later in the Chapter Macros
Inventory¶
The Inventory tab allows you to manually enter inventory information for the host. You can also select to enable Automatic inventory population, or disable inventory population for this host.
Encryption¶
The Encryption tab allows you to configure encrypted connections with the host. Il est possible de configurer l'encryption entre le Zabbix proxy/Zabbix server.
Value | Description |
---|---|
no encryption | (default value) Unencrypted |
PSK | Using pre-shared keys |
certificate | Using certificates |
Value Mapping¶
The Value mapping tab allows to configure human-friendly representation of item data in value mappings.
Host groups¶
Let's have look at the concepts of host groups and what the benefits are that
they provide.
We have seen that Host groups
can be created directly when we create
a new Zabbix host
.
Another way to create them is by a Super Admin
going to
Data collection
-> Host groups
.
Next press the button Create host group
in the upper right corner of the screen.
Host groups exists to make a logical group so we can add all hosts that belong
together in one group or more. Ex all Linux servers, all PostgreSQL servers,
or all the servers that belong to one team.
When going to our menu data collection
you notice that there are Host groups
and Template groups
.
If you come from an older Zabbix version you will be happy to read that Zabbix made
a specific group for Templates. If you are new to Zabbix don't panic :).
In older versions Zabbix had mixed Templates and host in one group.
This mixing was sometimes confusing especially for new users,
as Zabbix doesnt link templates to groups.
When you click on the menu Data collection
-> Host groups
.
You will notice that some groups are already made. You will also see that
there are some names behind the host groups with numbers in front.
These names are the names from the hosts that are in the group. The number in front
is the number of hosts that are in the host group.
To make life more easy you
can click on the names of the hosts and Zabbix will bring you directly to the
configuration screen for this host.
Zabbix allows the creation of nested groups. As you can see we are using forward slashes in our group name. When you make use of nested group you can use the '/' to separate groups.
Once our group or set of nested groups is made you can click again from the
host group
overview on the group.
You will notice that there is now a box that says
Apply permissions and tag filters to all subgroups
. When pressing this button,
all right that are this group will be applied to the sub-groups.
So if we have a user John
for example in a user group that has rights to see
everything in the Host group
with the name Europe/Belgium
and we apply
the option to the subgroups then our user John
will suddenly see also the hosts
in all our nestet groups and the tags on this host.
Note
When creating nested groups, Parent groups don't have to exist.
So we can have only the group open-future
without any of the parent groups.
It's up to the user to create them or not. Also group names cannot have /
in
their names. We cannot escape the /
character.
Also leading and trailing slashes and multiple slashes in a row are not allowed.
Tip
Have you tried to put emoticons in fields like host group yet ?
Interfaces¶
When setting up your host, if you want to monitor him, you will quickly need to specify an IP address or a FQDN.
This configuration is located in the interface section of the create host view.
After clicking add, a drop-down menu will ask you to select the type of interface.
Parameters | Description |
Agent | Allows use of item keys of type Zabbix Agent and Zabbix Agent Active |
SNMP | Allows monitoring of the host via SNMP type items |
JMX | Allows monitoring of Java applications through JMX counters |
IPMI | Allows monitoring of IPMI through items of type |
All types of interfaces need same information, IP or DNS and port number.
If you want to use DNSremember to click on the button DNS in Connect to
.
SNMP interface need four more informations:
- SNMP version, depends on the snmp version configured on the host
- SNMP Community, depend on the snmp community configured on the host.
- Max repetition count, necessary for native SNMP bulk requests Note that setting this value too high may cause the SNMP agent check timeout.
- Mark the Use combined requests checkbox to allow combined processing of SNMP requests (not related to native SNMP bulk requests)
You can create several interfaces on a host, which may or may not be identical. If more than one interface of a type exists, you can define the default interface (which will be used by default by all items on the host)
Note
We will cover IPMI in more detail later in the Chapter IPMI Monitoring We will cover JAVA in more detail later in the Chapter IPMI Monitoring We will cover SNMP in more detail later in the Chapter IPMI Monitoring
Tip
If you want to monitor host with HTTP call, interface are not necessary.
Templates¶
Templates are probably the most important notion in Zabbix, and globaly in monitoring tools. They help you to make change more easily and globaly, just make the change one time and will be applied immediately to all hosts concerned.
You can see template like a special host without interface that you can link to a classical host, the configuration added to a template are apply to all hosts using the template.
But in return, when you make a mistake during template evolution, it spreads to all hosts. Make little change and test on one or two hosts before if you have any doubts.
In this chapter we will teach you:
- How to add a new template
- Template configurations tabs
- Tags
- Macros
- Guidelines
How to add a new template¶
For build a new templates use the left menu and select Data Collection > Templates. On this menu you can see all templates available on you installation.
Before creating a new template you have to ask yourself one question : Does a template already exist for monitoring this technology ?
.
Of course the first way is to check the available template, and if no one exist you can check on : Zabbix Monitoring and Integration Solutions and more globally on the Internet.
Unfortunitely, no template exist for your needed. Use the Create template
button on the top right to start the creation.
Remerber the fields marked with a red asterisk "*" are the fields that are mandatory.
A template is simply a name and a one or more template groups.
Tip
Suffix the technonology name with the type of item used by Zabbix agent
, by SNMP
, ...
Template groups is like hostgroup, it's use for classify template and apply rights.
In description you can add information that could be of interest to use the template, the version tested, ...
Tags¶
Template-level tags will be visible in case of problems.
Macros¶
All user macros used by the template need to be define with default value in this section. Your template must be able to use without setting on host, except for connection informations, the default value especially the thresold must be generic.
Some useful links¶
https://www.zabbix.com/documentation/guidelines/en/thosts/configuration/templates
Items¶
Macros¶
Macros are one of the concepts you need to understand in Zabbix. There are different types, and they make Zabbix more scalable and less static. In this topic we will explain the different types of macros and how to use them.
We will cover:
- Built-in macros
- Function macros
- User macros
- Low-level discovery macros
- Expression macro
- User macro with context
Built-in macros¶
They correspond to Zabbix internal macros, they are represented using this
notation {MACRO}
, the complete list is available here.
Each built-in macro can be used in one or more cases.
For example, the {HOST.HOST}
built-in macro can be used to retrieve the host
in a trigger expression, or the macro {HOST.IP}
which does the same thing
but retrieves the IP of the host.
Function macros¶
You may want to manipulate the value of a macro, to shorten or extract a part
for example. The syntax of a macro function is as follows: {macro.func(params)}
Function | Description |
---|---|
fmtnum | Number formatting to control the number of digits printed after the decimal point. |
fmttime | Time formatting. |
iregsub | Substring extraction by a regular expression match (case-insensitive). |
regsub | Substring extraction by a regular expression match (case-sensitive). |
User macros¶
Macros are what give Zabbix its great flexibility. They can be defined globally,
or at template or host level. They are always preceded by a $ and are
noted thus: {$MACRO}
.
It's important to note that Zabbix prioritises user macros in the following way:
- Host level macros (checked first)
- Macros defined on first level templates of the host (i.e., templates linked directly to the host), sorted by template ID
- Macros defined on the second level templates of the host, sorted by template ID
- Macros defined on the third level templates of the host, sorted by template ID, etc.
- global macros (checked last)
These macros can be used in many locations, they are commonly used to define a threshold and serve as a default value in a template that can be overloaded with host-level macros.
In order to have templates that are reusable and functional in the majority of the situations, users macros must be used.
It's also recommended to not use global macros when not needed as they are not portable when exporting hosts or templates.
So when we define global macros
they will be overwritten by template macros
and our template macros will be overwritten by host macros
.
It's easy so overwrite a macro by accident to solve this problem Zabbix has
the option on Host
and on Template
level to see if there are any macros
defined yet with the same name. This makes it easy to see if you have this
macro defined already and if it will be overwritten.
In this screenshot we can see that there is a Global macro
defined with
a value of 100 and that we have the same macro on Template level
that overrides
the value of the Global macro
. However the effective vale is coming from
our Host macro
.
Low-level discovery macros¶
This type of macro is a little special, it is used in the case of Low-level discovery
and is preceded by a # {#MACRO}
. It's useful for the creation of a
prototype item, prototype trigger or prototype host.
Expression macro¶
These macros are very specific, they are used exclusively in name graphs and in maps,
they use the following syntax: {?EXPRESSION}
. {?avg(/{HOST.HOST}/{ITEM.KEY},1h)}
User macro with context¶
There's one last subject concerning macros, and it's not the least important:
very soon after using your first user macros to define thresholds, you'll realise
that you want to have a different threshold depending on ... the context.
For this we can use the following syntax: {$MACRO: "static text"}
If you want to set the global threshold to 80%, but on a particular partition
you want to increase the threshold to 90%. To do this, we'll use the context.
Example¶
- {$LOW_SPACE_LIMIT}: 80
- {$LOW_SPACE_LIMIT:/tmp}:90
In this example, all the mount points on our server will have a default threshold of 80% except for /tmp, which will have a custom threshold of 90%.
Some useful links¶
- https://www.zabbix.com/documentation/7.0/en/manual/appendix/macros/supported_by_location
- https://www.zabbix.com/documentation/7.0/en/manual/config/macros/macro_functions
- https://www.zabbix.com/documentation/7.0/en/manual/config/macros/user_macros
- https://www.zabbix.com/documentation/7.0/en/manual/config/macros/lld_macros
- https://www.zabbix.com/documentation/7.0/en/manual/config/macros/expression_macros
- https://www.zabbix.com/documentation/7.0/en/manual/config/macros/user_macros_context
Data Flow¶
Data Collection¶
Simple Checks¶
Ended: The basics
Data collection ↵
Zabbix Agent¶
In this topic we will talk about the Zabbix agent. The Zabbix agent is a native agent made by Zabbix that we can deploy on different Operating Systems and that supports a wide range of build-in items keys.
Zabbix has 2 agents we have the Zabbix agent
and we have the Zabbix agent 2
So when you read in this book about the Zabbix agent
we talk about the 1st agent. This can sound a bit
weird having 2 agents but it's not. The agent was developed years ago when zabbix was first released and is written in C the agent 2 is written in GO and is developed next to the Agent 1.
We will see later what the advantages and disadvantages are of both and give you tips to choose the agent that fits best for you.
To make things more complex both agents can be configure to work in active
or in passive
mode or in both.
When we create an item on our host and we want to use the item that works with our Zabbix agent we have 2 options to choose from.
We can create items that are active or passive depending on how our agent has been configured. A zabbix agent can either be active or passive or both active and passive.
Note
Zabbix 7.0 is backwards compatible with agents back to 1.4 but of course newer item keys added after 1.4 will not be recognized in those older agents. So if you like to use the new items keys it's best to upgrade your agents when you can.
Sometimes there are cases when the item keys provided by Zabbix are just not enough.
In this case it's possible to extend our agents with UserParameters.
This allows us to extend the agent with our own keys based on scripts that we make to extend our agents. This is something we cover in the topic UserParameters
.
Zabbix has a huge list of items keys it supports on linux but when we select our item keys we have to careful read the documentation careful as some items keys are OS specific. For example there are item keys that only work on MS Windows.
https://www.zabbix.com/documentation/7.0/en/manual/config/items/itemtypes/zabbix_agent/win_keys
Some item keys can only be used on the new Agent 2.
https://www.zabbix.com/documentation/7.0/en/manual/config/items/itemtypes/zabbix_agent/zabbix_agent2
Here is a list of the item keys that will work on Linux. But also here be careful as some things can be different depending on the OS or the kernel version.
https://www.zabbix.com/documentation/7.0/en/manual/config/items/itemtypes/zabbix_agent
When we want to add an item we can either lookup the item in the online documetation, try to remember it and just write it in the box or press the select button next to the item field. This will give a popup with a short explanation about what the items does.
However you will see that it's lacking lots of information. My advice is to use the online documentation.
So in this topic we will explain you all the differences :
- Zabbix Agent (Legacy)
- Zabbix Agent 2
- Agent vs Agent 2
- Active Agent
- Passive Agent
- Active vs Passive
- Zabbix agent Linux
- Zabbix agent Windows
Zabbix Agent (Legacy)¶
Our Zabbix agent or the legacy agent can run on a whole set of supported platforms. At the time of writing the agent on Zabbix 7 supports:
- Linux (Redhat, Suse, Ubuntu, Debian, Rocky, Alma, ...)
- FreeBSD
- Windows
- MacOS
- Raspberry Pi
- HPUnix
- OpenBSD
- AIX
- Solaris
Info
A full up to date list can be found here https://www.zabbix.com/download
The agent can be installed on a host to collect data from that host or from an application running on that host. The advantage is that the agent has a very small footprint and that we can extend the agent by creating our own scripts. Agents can work in active or in passive mode or can do this in both ways simultaniously.
The legacy agent can be extended with modules that are written in C. More information about modues can be found in the Zabbix documentation https://www.zabbix.com/documentation/7.0/en/manual/extensions/loadablemodules.
The Zabbix agent will run with limited privileges and user zabbix
. With the legacy agent this can be changed easy if you want.
In the config file there is an option User=zabbix
that can be changed. This however will not work if you are like 99% running on an OS that uses SystemD. Same it's not possible to run the agent with root
privileges by enabeling the option AllowRoot
. Changing these options will only work on non SystemD systems.
What we need to do is change the SystemD service for our zabbix-agent
.
systemctl edit zabbix-agent
Next we have to add the following content. If you like to run as another user then root then just create an account and replace root with this account.
[Service]
User=root
Group=root
Exit the file and save it. We now have to tell SystemD to reload the config by running the next command.
systemctl daemon-reload
The only thing that rests us now is restarting the zabbix-agent
so that it picks up the config changes.
systemctl restart zabbix-agent
Warning
Running the Zabbix agent under root privileges is not recommended. It's a high security risk but in some cases it is maybe needed and then this is your only solution.
Note
The agent will inculde by default all the files under /etc/zabbix/zabbix_agentd.d/ with extension .conf. Its good practice to not edit the standard config file and just add your changes in a dedicated file in this folder.
Zabbix Agent 2¶
The Zabbix Agent2 is the new generation of agent and is written in GO. Because of this i's not possible to install the Agent 2 on systems like BSD or HPUniX. At the moment there are only clients for linux and Windows.
Since
Info
A full up to date list can be found here https://www.zabbix.com/download If your OS supports GO and there are no packages you could always try to download the source and compile it by your own.
Our agent 2 can be extended with plugins just like the Agent was extensible with modules. However plugins are way more advanced then the modules on agent 1 and there is already an extensive list of modules that are supported. The full list can be found here https://www.zabbix.com/documentation/7.0/en/manual/appendix/config/zabbix_agent2_plugins.
Running the Agent2 under root or another non privilege account is more or less the same as with the legacy agent. This agent only runs on Linux systems with SystemD.
systemctl edit zabbix-agent2
Next we have to add the following content. If you like to run as another user then root then just create an account and replace root with this account.
[Service]
User=root
Group=root
Exit the file and save it. We now have to tell SystemD to reload the config by running the next command.
systemctl daemon-reload
The only thing that rests us now is restarting the zabbix-agent
so that it picks up the config changes.
systemctl restart zabbix-agent2
The plugins have their own timeout settings and the agent will do a compatibility check when it starts up.
Another advantage of the agent2 is that it supports scheduled and flexible intervals for active and for passive checks. Something that is not possible with the agent.
Plugins can execute tasks in parallel and scaling up to 1000 workers per plugin. Every plugin has it's own set of configuration parameters. The syntax used for this is Plugins.<PluginName>.<Parameter>=<Value>. As example the plugin to configure the logging of remote commands looks like
Plugins.SystemRun.LogRemoteCommands=0```.
Some of the plugins even work with together with templates. This makes it easy so configure some settings by changing macro values in the templates. Some more advanced parameters can then still be adjusted in the config of the plugin.
Warning
Running the Zabbix agent under root privileges is not recommended. It's a high security risk but in some cases it is maybe needed and then this is your only solution.
Note
The agent will inculde by default all the files under /etc/zabbix/zabbix_agent2.d/ with extension .conf. Its good practice to not edit the standard config file and just add your changes in a dedicated file in this folder.
Agent1 vs Agent2¶
Parameter | Agent | Agent 2 |
---|---|---|
Programming language | C | Go and some parts in C |
Linux daemonization | Yes | By systemd only |
Run as Windows service | Yes | Yes |
Supported extensions | Loadable Modules in C | Plugins in Go |
Supported platforms | All | Linux, Window, any OS with GO |
Concurrency | Active checks are executed sequentialy | All check executed concurrently |
Scheduled/flexible intervals | Passive check only | Acive and Passsive checks |
Persistent storage | No | Yes (SQLite3) |
Timeout settings | On agent level only | Plugins can override the timout |
Changes user at runtime | Yes (on Unix like sytems only) | No (controlled by SystemD |
Cipersuits user configurable | Yes | No |
Info
For more details bout the differences check the official documentation. https://www.zabbix.com/documentation/6.0/en/manual/appendix/agent_comparison
We can detect the generation from the agent by using the agent.variant
item key this will return us 1 for the legacy agent and 2 for agent2.
agent.variant
Active Agent¶
Passive Agent¶
Active vs Passive¶
UserParameters¶
Zabbix agent Linux¶
The agent can be installed on Linux in different ways. The most easy way however is probably installing the agent from Package.
Most of the popular operating systems are supported by Zabbix and installation instrucations are provided on the Zabbix webpage when you go to https://www.zabbix.com/download
Here we just have to choose that we like to install from Zabbix packages and choose our OS and the version. Then we just need to follow the instructions to add the repository to our OS. From here we can install the Agent.
After selecting the correct settings we get to see the instructions how to add our repo to the OS.
In case your OS was not in the list there is still the option to download the agents pre compiled. Choose Zabbix agents
and select your correct kernel version.
Then select the Zabbix release and press Download
to get a binary version.
If all this fails then there is still the option to download the source and compile the agent yourself.
The Agent can be installed from the reposity with dnf install zabbix-agent
after the installation the configuration files can be found at /etc/zabbix/zabbix_agentd.conf
.
The log files will be written in /var/log/zabbix_agentd.log
by default.
The Agent2 can be installed from our repository with dnf install zabbix-agent2
.
After the installation, the configuration files can be found at /etc/zabbix/zabbix_agent2.conf
.
The log files will be written in /var/log/zabbix_agent2.log
by default.
Zabbix agent windows¶
Ended: Data collection
Problem detection ↵
Triggers¶
Ended: Problem detection
Taking action when problems come ↵
Event based Actions¶
Ended: Taking action when problems come
Managing permissions ↵
Managing Permissions¶
Ended: Managing permissions
Visualising Problems ↵
Visualising our problems¶
Ended: Visualising Problems
Automating configuration ↵
Automating configuration¶
Ended: Automating configuration
VMWare monitoring ↵
VMware monitoring with Zabbix¶
Zabbix can automatically discover Virtual Machines and VMware hypervisors based on Low-level discovery rules. Zabbix provides a set of templates to do this out of the box to make life more easy. In this chapter, we will see how we can configure this.
Zabbix requires VMware vCenter or vSphere to be at least 5.1 or higher but always check the official documentation to be sure this hasn't changed.
What do we need ?¶
- VMware vCenter or vSphere to be at least 5.1
- A monitoring account with RO access to the API
- Enable performance counter statistics and requests on the VMware side as well.
Before we start it's important to know that Zabbix does not start any processes to start monitoring VMware out of the box if we do not configure this.
So the first thing we have to do is edit out zabbix_server.conf
file. This file can be found here : /etc/zabbix/zabbix_server.conf
.
Look for the parameter StartVMwareCollectors, edit this line, or add a new line where you add StartVMwareCollectors=2
and restart the server service.
Why do we add 2? Is it always 2? Of course not else it would be hardcoded. But Zabbix recommends not going below 2 and it should also not exceed twice the amount of monitored VMware services.
So we can use the next formula to calculate the optimal value?
Amount of services < StartVMwareCollectors < (Amount of services * 2)
So for the setups with 1 VMware service we will set it to 2 and with instances where we have 3 we will set it to 5
Note
A VMware instance can be a vSphere or vCenter instance and not a virtual machine or hypervisor. So the number of collectors depends on the data points we use to collect values from.
Configuration¶
The first step is to create a new host. If you are unsure how to take a look at the topic Zabbix hosts.
Only adding a Host name
should be ok we don't need to configure any interfaces. Once this is done before you press Add
go to the tab Macros
.
Add the following macros :
- {$VMWARE.USERNAME}
- {$VMWARE.PASSWORD}
- {$VMWARE.URL}
Tip
Do not use the admin account of your VMware VCenter but create a new monitoring account with RO rights to read out the information and use a secret macro for the password and even maybe the username.
Once done go back to the Host
tab and fill in the Host group
and the correct template.
Zabbix provides 2 templates to choose from.
- VMware FQDN : The template making use of the FQDN for the VMs (recommended) but:
- Every Monitored VM should have a unique OS name compliant with the FQDN rules.
-
VMware Tools must be installed on every machine
-
VMware : The template making use of the UUID for every VM.
-
This one is recommended if the FQDN requirements cannot be met.
-
VMWare Guest : Will monitor the virtual machines
- VMWare Hypervisor : Will monitor the hypervisors
- These 2 templates should not be linked manually Zabbix will handle this for you !
Zabbix will discover VMware clusters, hypervisors, data stores, and virtual machines. It's possible to control which parts will be monitored by going to the template to the Discovery and disabling the Discovery rules you don't need.
It's also possible to change the configuration of our host prototype. Ex: we can configure the host interface to Inherit
or to create the interface with Custom
data.
A list with MACROS
can be found here https://www.zabbix.com/documentation/7.0/en/manual/vm_monitoring/discovery_fields
As you have may have noticed already, in the Zabbix server configuration file there are a few more parameter that we can tweak.
- VMWareFrequency
- How often Zabbix will connect to the VMWare service to collect new data.
- VMwarePerfFrequency
- How often Zabbix will connect to the VMWare service to collect performance data.
- VMWareTimeout
- This is the maximum number of seconds that a VMWare collector will wait for a response from the VMWare vCenter or Hypervisor
- VMwareCacheSize
- This is the shared memory size for storing VMware data. A VMware internal check zabbix[vmware,buffer,...] can be used to monitor the VMware cache usage. This should be verified if enabled in your
zabbix server healt template
. The size of the cache will depend on the size of your VMware setup.
Note
It's best to keep VMWareFrequency and VMwarePerfFrequency with the same values. Using values where VMWareFrequency
is less then VMwarePerfFrequency
will lead to strange looking graphs.
Note
If config.vpxd.stats.maxQueryMetrics is invalid or exceeds the maximum number of characters permitted error, add a config.vpxd.stats.maxQueryMetrics parameter to the vCenter Server settings. The value of this parameter should be the same as the value of maxQuerysize in VMware's web.xml file.
Available metrics¶
After some time in latest data you should see for all the items data being populated. It can take a while before every item is populated it depends a bit on the order of the LLD rule being executed.
These are the items that are configure with our standard Zabbix VMware template. But the template does not cover all items and more items might be added over time. A list of all keys can be found in the online documentation. https://www.zabbix.com/documentation/7.0/en/manual/vm_monitoring/vmware_keys
Those items are simple checks
so when creating a new item don't forget to select the correct item type.
Internal working¶
We know now that Zabbix can monitor VMware out of the box and the setup for it is very easy. There is a lot of information that we can retreive but to get to this information there is a complex process behind it. To tune this process we have seen that there are parameters that can be set like VMWareFrequency, VMwarePerfFrequency, VMWareTimeout and VMwareCacheSize.
The next diagram shows us how Zabbix collects the data and end up with the data in it's history Cache.
Here we can see the different steps involving in the data flow when monitoring VMWare with Zabbix. Connections are made from the VMWare collector to the SDK interface. The collectors will grab the data and place it in a special VMWare chache. Pollers then will look into the cache and grab the data from it and send it to the preprocessors. The data after preprocessin is then placed in the Zabbix history cache. In case of a proxy it will be sent to the Zabbix server.
Note
VMwarePerfFrequency and VMWareFrequency are both set standard to 60 seconds wich is fine for small to medium setups lowering this number could create some overload on the VMWare servers. On a larger instance it's probably best to increase then together with the VMWareTimeout parameter.
Debugging¶
Warning
It can take some time before you will see all the VMs in your Zabbix setup popup. If you want to know what the Zabbix server or Proxy (if you monitor VMware over a proxy then you have to adjust all config on the proxy) is doing then you can increase the debug level. This can be done on they fly without changing the debug parameter in the zabbix_server.conf file. Just one the console run the following command to increase logging for all the vmware collectors.
zabbix_server -R log_level_increase="vmware collector"
In case you like to increase logging for only collector 2 then you could run the following command:
zabbix_server -R log_level_increase="vmware collector,2"
Note
If you see a message like "config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted**" in the debug don't be alarmed right away. The message is a bit confusing. Have alook at this ticket as it will explain that its not always a problem. https://support.zabbix.com/browse/ZBX-24499
Ended: VMWare monitoring
Monitoring websites ↵
Monitoring websites¶
Ended: Monitoring websites
Monitoring SNMP, IPMI and JAVA ↵
SNMP Monitoring¶
In this topic we will explain you how to monitor devices over SNMP.
SNMP trap monitoring¶
If you have been reading previous topic about SMNP monitoring then you know by now that SNMP monitoring is all about polling information form a device. However there is another way to gather information from a device with SNMP that does not expect us to poll for information but that accepts information being sent from the SNMP device to our Zabbix server or Proxy. This is what we call SNMP traps.
Traps are sent from a device to our Zabbix server or proxy and usually are only being sent when there is a change in the status of a device. Traps also dont use TCP but are being sent over UDP to port 162. So why would we use traps as we already can poll SNMP devices ?
Well first of all let's imagine there is a short fluke in the power of your switch for only a second. This would never be noticed by Zabbx as we usually poll an item once per minute or even less. If we had to poll every second it would create a huge load on the device and we would maybe overload it and we would need also a more beefy Zabbix server if we had to to this for every item.
A trap is sent when a problem is detected so this is extra data that else would probably be missed by Zabbix. There are of course some disadvantages with traps else we would only use traps and never poll for problems. The problem with traps is that they are sent over UDP and the problem with UDP is that packages are being sent without waiting for any acknowledgement that they have arrived at the destination. It's a bit like talking to your wife not even knowing she is in the same room and then thinking she has understood you.
We all know how that works out ....
So the best way to monitor SNMP devices is probably an implementation of both SNMP polling and SNMP traps.
There is one small problem that is not really a problem but complicates things a bit and that is that Zabbix has no build in trap receiver. Zabbix can use SNMP traps but it needs to rely on the OS for this and so we need to install snmptrapd to catch traps when they are bein sent to our Zabbix server/proxy .
Once we have catched the trap we need to process the trap to make it readable by Zabbix and for this there are 2 ways:
- We can make use of a script usually a perl script that is delivered with the sourcecode from Zabbix or a bash script
- by making use of SNMPTT.
Since there are 2 solutions it makes sense that they both have advantages and disadvantages as you probably already know.
- The perl/bash script is the most easy solution and quit performant but more difficult to make customizations so less flexible.
- The SNMPTT,snmptrapfmt, ... is more complex but give more flexiblity.
This table will give us a short overview of the differences between the 2 solutions:
Perl Script | SNMPTT | |
---|---|---|
MIBS | Not Required | Required |
Trap formatting | Script | Config file |
Trap matching | snmptrap["PCRE"] | snmptrap["PCRE"] |
Unknown traps | snmptrap.fallback | Configuration file |
Acc or reject trap | No | Yes |
Search and replace | No | Yes |
The perl script is the most easy way so if you don't need any of the fancy features that are not possible with the perl script it's probably best to stick to this script. In this book we will cover the script. The other way is more or less the same but you will have to do some tuning yourself on the config of SNMPTT or other solution.
The script can be found in the source code. You can download the source here. Once downloaded you can find the perl script in the folder /misc/snmptrap/zabbix_trap_receiver.pl
If you can not use perl you can find the bash script in the same location.
Setting up snmptraps¶
Now that we know a bit more about snmptraps let's set it up and configure our Zabbix server. Traps are sent to the zabbix server so in case we have a proxy insteqd of the zabbix server we need to send traps to the proxy.
Warning
At this moment Zabbix has loadbalancing for proxies implemented but there is no support for SNMP traps in this case. So only the active proxy will accept traps.
Configure SNMP traps on our system with snmptrapd¶
First thing first we will have to open our firewall if we want to accept traps. Traps are being sent over UDP and arrive on port 162 so let's open a port.
# firewall-cmd --add-port=162/udp --permanent
# firewall-cmd --reload
Next we need to install the net-snmp, net-snmp-utils package and the net-snmp-perl package for our perl script to work with snmptrapd.
dnf install -y net-snmp net-snmp-utils net-snmp-perl
Copy our zabbix_trap_receiver file to /usr/bin/ and make it executable
# cp zabbix_trap_receiver.pl /usr/bin/.
# chmod +x /usr/bin/zabbix_trap_receiver.pl
We can now configure snmptrapd
and tell our server what traps it should accept.
# vi /etc/snmp/snmptrapd.conf
add the following line
authCommunity execute public
perl do "/usr/bin/zabbix_trap_receiver.pl";
This line will tell our snmptrapd
to accept all the traps that are being sent to our device with the community string set as public. so please adapt it to your needs.
We also need to tell our Zabbix server to start up the trapper process this can be done in the configuration file of the server. Edit the following file
# vi /etc/zabbix/zabbix_server.conf
And look for the line with the parameter StartSNMPTrapper=
and make sure that the line is active by removing the # in front and replace 0
with 1
so that Zabbix knows it needs to activate the trapper for SNMP
StartSNMPTrapper=1
There is another parameter that we need to configure and that is SNMPTrapperFile=
. Same here make sure that the line is active by removing the #
in front and poit to the location of the location where Zabbix can read the traps that are written by our perl script.
If it's not the same then adapt the script or zabbix_server.conf file so thqt in both files the location point to the same path: The perl script will write with trap information in this file and Zabbix server will look for this file to read the information.
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
Make sure the folder exists
mkidr /var/log/snmptrap/
Once everything is done restart the Zabbix server so that it picks up the config changes.
# systemct restart zabbix-server
Also enable snmptrapd
and make sure it starts at boot.
# systemctl enable snmptrapd --now
There is one thing we need to do extra and that is to configure log rotate. Snmptrapd will sent traps to the /var/log/zabbix_traps.tmp
file and the file weill keep growing and growing so we need to make sure logrotate will cleanup from time to time.
Zabbix is not managing the trap file so we need to create the logrotate config for our trap file else it will keep growing over time
# vi /etc/logrotate.d/zabbix_traps
Copy following content in to this file and adqpt to your own needs.
/var/log/snmptrap/snmptrap.log {
weekly
size 10M
compress
notifempty
dateext
dateformat -%Y%m%d
olddir /var/log/snmptrap/
maxage 365
rotate 10
}
Testing if traps are received¶
To see if we can receive traps let's run a few tests on our system. In case things don't work as expected verify every step again see that snmptrapd is calling the perl script on the correct location ,verify that your community names are correct, make sure snmptraps are active in the Zabbix server configureation. And in your perl script check that file will be written to the correct folder and that the script is executable.
- # snmptrap -v 1 -c public 127.0.0.1 '.1.3.6.1.6.3.1.1.5.4' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.4 s "enp0s1"
If all goes well we should now see in our log file the trap
# cat snmptrap.log
2024-06-08T19:03:08+0200 ZBXTRAP 127.0.0.1
PDU INFO:
messageid 0
requestid 0
version 0
community public
notificationtype TRAP
errorstatus 0
receivedfrom UDP: [127.0.0.1]:32790->[127.0.0.1]:162
errorindex 0
transactionid 5
VARBINDS:
DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (55) 0:00:00.55
SNMPv2-MIB::snmpTrapOID.0 type=6 value=OID: IF-MIB::linkUp.0.33
IF-MIB::linkUp type=4 value=STRING: "enp0s1"
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 type=4 value=STRING: "public"
SNMPv2-MIB::snmpTrapEnterprise.0 type=6 value=OID: IF-MIB::linkUp
We can now do the same for a snmp trap v2
- # snmptrap -v 2c -c public localhost '' 1.3.6.1.4.1.8072.2.3.0.1 1.3.6.1.4.1.8072.2.3.2.1 i 123456
This should return the following information
# cat snmptrap.log
2024-06-08T19:03:08+0200 ZBXTRAP 127.0.0.1
PDU INFO:
messageid 0
requestid 0
version 0
community public
notificationtype TRAP
errorstatus 0
receivedfrom UDP: [127.0.0.1]:32790->[127.0.0.1]:162
errorindex 0
transactionid 5
VARBINDS:
DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (55) 0:00:00.55
SNMPv2-MIB::snmpTrapOID.0 type=6 value=OID: IF-MIB::linkUp.0.33
IF-MIB::linkUp type=4 value=STRING: "enp0s1"
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 type=4 value=STRING: "public"
SNMPv2-MIB::snmpTrapEnterprise.0 type=6 value=OID: IF-MIB::linkUp
2024-06-08T19:05:31+0200 ZBXTRAP 127.0.0.1
PDU INFO:
notificationtype TRAP
community public
messageid 0
requestid 2002881661
version 1
transactionid 6
errorindex 0
receivedfrom UDP: [127.0.0.1]:53810->[127.0.0.1]:162
errorstatus 0
VARBINDS:
DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (218161) 0:36:21.61
SNMPv2-MIB::snmpTrapOID.0 type=6 value=OID: NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification
NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate type=2 value=INTEGER: 123456
First trap has sent us the name of our networkcard the second trap has sent us the value 123456
.
Create a trap item in Zabbix¶
So we have our traps working but there is still something missing. We don't see any traps yet in Zabbix. This is because we have not created an item yet. We still have to tell zabbix to look in our trap file for items that we like to monitor.
Zabbix can monitor for specific items like for example the trap with the networkcard we just sent or in case we don't know what exact item will arrive Zabbix has also a fallback option. Let's create both items so that we have an idea how to do this.
Create a fallback item¶
Create a specific item¶
Before we can create an item we have to add a SNMP
host interface on our host. Go to Data collection
-> Host
click on the host where you would like to sent the trap to and add a SNMP
host interface with the correct IP of your host. In my case I am sending a trap to the zabbix server with the loopback interface as the IP. This will be different for every host. Zabbix will look in your log file and match the IP in the log with the host in your Zabbix setup.
Also for this to work you will have to disable SeLinux
as it will block or Zabbix server from accessing the log file. So don't forget to fix this afterwards.
to temporary disable run setenforce 0
.
Once you have added your SNMP interface go to Data collection
-> Hosts
and click on items behind the name of your hosts.
Let's create or fallback item by clicking on Create item
in the upper right corner.
When you see the popup to create the new item we have to add a few things copy everything I will explain later what it does.
Name : SNMP Fallback
Type : SNMP trap
Key : snmptrap.fallback
Type of information : Numeric
Host interface :
Go to the tab Preporcessing and fill in the following information for our Preprocessing setp 1:
- Name: Regular expression
- Parameters: INTEGER:.(\d+)
- after the parameter field there is another field here you can fill in : \1
So let us explain quick what we just filled in.
The name is just the name that will be visible in our latest data page etc for our item. The type we have to select SNMP trap
as we are monitoring for incoming SNMP traps.
The key is freeform and needs to be unique so we just choose something that makes sense in this case snmptrap.fallback
as this item is a catch all item.
Type of information I have chosen numeric usually you will want to keep it as text. I have chosen numeric because with preprocessing I will extract a numeric value from our test trap.
In the Preprocessing tab
We added a regular expression in Perl this expression will look for the line starting with INTEGER:
and the numeric value behind it. by adding ( ) we created a group. the \1 is a selector for the number of groups we like to extract in this case we have only 1 group so we look for group 1
Sent the following trap to your zabbix server and replace the IP with the IP of the hosts you have added.
snmptrap -v 2c -c public 127.0.0.1 '' 1.3.6.1.4.1.8072.2.3.0.1 1.3.6.1.4.1.8072.2.3.2.1 i 123456
Just run the trap on your Zabbix server and have a look in the latest data page of your hosts
Monitoring -> Latest data and fill the name in of your host
in the Hosts
field.
In the Name
field you can filter for fallback if there are too many items
Adding a specific SNMP trap item works exactly the same only here we dont use the item key snmptrap.fallback
but the item key snmptrap[regex]
.
So in our case it will be snmptrap[123456].
Create the same item as before but don't add any Preprocessing rules this time.
Let's send the trap again and go to latest data to see what we have received this time.
snmptrap -v 2c -c public 127.0.0.1 '' 1.3.6.1.4.1.8072.2.3.0.1 1.3.6.1.4.1.8072.2.3.2.1 i 123456
It looks like something went wrong in latest data we got a warning. This warning as in the screenshot above can be seen when you go with your mouse over the i
in red at the end of you item.
So what happened here is that Zabbix looks for a regex in this case 123456
it has found in the trap the value 123456
and it will show us the complete trap.
So we have 2 options here :
- We can change the item it's
type of information
toText
and choose to see the whole trap. or - We can add again a Preprocessing step like we did with our fallback and filter for the numbers
123456
.
Once this is done our items will work.
Working with SNMPv3 Traps¶
If you like to use SNMPv3 devices and protect the traps that are being sent then this is also possible. It works like with SNMPv1 and v2 we just have to change the config in our snmptrapd config and add the following lines.
createUser -e <engineid> <user> SHA <key> AES <key>
authUser log,execute <user>
perl do "/usr/bin/zabbix_trap_receiver.pl";
This will make sure snmptrapd will catch our SNMPv3 traps and sent them to our perl script.
Note
An engine ID should be unique for every device you can't have devices with the same engineid
so for every device you will have to add a line in the config file.
Tip
The PDU info can be removed from the zabbix_trap_reciever.pl in case you dont like it
Remove the folowing lines in the perl script or add a #
in front :
# print the PDU info
print OUTPUT_FILE "PDU INFO:\n";
foreach my $key(keys(%pdu_info))
{
if ($pdu_info{$key} !~ /^[[:print:]]*$/)
{
my $OctetAsHex = unpack('H*', $pdu_info{$key}); # convert octet string to hex
$pdu_info{$key} = "0x$OctetAsHex"; # apply 0x prefix for consistency
}
printf OUTPUT_FILE " %-30s %s\n", $key, $pdu_info{$key};
}
Some useful links¶
- https://www.unix.com/man-page/redhat/8/logrotate/
- https://www.netreo.com/blog/snmp-traps-definition-types-examples-best-practices/
- https://www.zabbix.com/documentation/7.0/en/manual/config/items/itemtypes/snmptrap
- https://net-snmp.sourceforge.io/wiki/index.php/TUT:Configuring_snmptrapd
- https://net-snmp.sourceforge.io/wiki/index.php/TUT:Configuring_snmptrapd_to_receive_SNMPv3_notifications
IPMI Monitoring¶
We have already seen a few ways to monitor with Zabbix what we havent seen yet is monitoring with IPMI.
IPMI stands for Intelligent Platform Management Interface
and is usually an extra interface in your hardware that needs to be added.
It's a set of standard specification for hardware platforms that allows to control and monitor the hardware.
An IPMI system has several key components:
- BMC (Baseboard management controller) : This is a microcontroller that monitors and manages the hardware of the system independently even when the server is offline.
- IPMB (Intelligent platform management bus) : This is a framework that allows interactions between all the IPMI components. This is the most important part for us as it has a key role in system monitoring by supporting sensor data collection and the execution of control commands.
- IPMI Memory : All sensor data, configuration settings and event logs will be stored in memory to make it easier to run diagnostics during failures when the server is offline.
- Dedicated management network : This network is designed explicitly for IPMI traffic, providing a reliable channel of communication with the BMC.
The features that are provided by IPMI are:
- Out-of-band management: A feature that enables the remote management of servers independently of the operating system and it's status. It is neededfor performing firmware updates, troubleshooting, and executing recovery procedures without the need for direct physical access to the hardware.
- Hardware monitoring : IPMI provides real-time information of critical system parameters, like temperature, voltage, fan speed, power consumption, disk health, .... . By monitoring these metrics, IPMI can help to quicklyidentify and address potential issues before they escalate.
- Event logging: The System Event Log (SEL), records system events, this will offering valuable resources for post-event analysis and troubleshooting.
- Remote power control: This offers administrators control over the equipment's power state from a distance. It maes it easy to turn devices on/off, performing resets, and cycling power for system management and recovery.
- Virtual media: IPMI enables mounting remote storage devices as though they were physically connected to the server, this makes remote installations more easy.
- Security features: IPMI includes security to authenticate user identity and control access permissions. This ensures only authorized users can access.
Note
When writing, the landscape of IPMI is defined by two prominent versions: 1.5 and 2.0. Although both versions are in active use, IPMI 2.0 is distinguished by its advanced features, heightened security measures, and superior performance, making it the preferred option for modern server management.
In this chapter you will learn how to :
- Configuring an IPMI device
- Configure Zabbix to monitor an IPMI device
- Setting up an IPMI item
- Configure LLD with IPMI
- Monitor discrete sensors
Configuring an IPMI device¶
Usually you need some hardware device with an IPMI interface. This can be for example a iDrac interface from DELL or an ILO interface from HP. Every hardware vendor has some card based on IPMI but names can be vendor specific
If you don't have a hardware device that supports open IPMI don't worry you can install an ipmi simulator. It does not provide any network information but provides the basic functionality to play with.
To Install the container run the following commands:
sudo dnf install podman
sudo podman pull vaporio/ipmi-simulator
sudo podman run --name ipmi_simulator --publish 623:623/udp docker.io/vaporio/ipmi-simulator:latest
sudo dnf install ipmitool
Once done everything should be up and running and we can test to see if we can connect to it .
ipmitool -H 127.0.0.1 -U ADMIN -I lanplus -P ADMIN chassis status
if you like to do it from another VM that is also possible we will have to do it later from our Zabbix server anyway but then don't forget to open the firewall port 623/udp.
firewall-cmd --add-port=623/udp --permanent
firewall-cmmd --reload
Note
It's good to know that IPMI is not free from security problems. Intel , Dell, HP , ... even encourage not to use it and we should not expect any updates on the IPMI specifications in the future. https://www.intel.com/content/www/us/en/products/docs/servers/ipmi/ipmi-home.html?wapkw=ipmi
Configure Zabbix to monitor an IPMI device¶
Before we can start configuring items in Zabbix we need to configure the Zabbix server to start some IPMI pollers. This can be done in the zabbix_server.conf
file.
Zabbix relies on the OpenIPMI library to monitor IPMI. So to get things started let's edit /etc/zabbix/zabbix_server.conf
and change the line # StartIPMIPollers=0
.
Setting up an IPMI items¶
Before we can dive into Zabbix and start adding items we need to be sure we have access to our IPMI. IPMI makes use of port 623 and works on top of UDP, so it's in the OSI 5-7 range so we need to be sure there is no firewall blocking this port.
Make sure you have enough pollers in the zabbix_server.conf
for example StartIPMIPollers=3
and dont forget to restart the server.
We can test the access to the IPMI console with the following command:
ipmitool -U <User> -H <IP address of the IPMI host> -I lanplus -P <password> -L user sensor
The -L is to give user access so that we dont need any administrative privileges. The lanplus
option instructs our ipmitool to use the IPMI 2.0 Lan interface and the sensor queries the host for a list of available sensors. You could launch the command without the word sensor then the command will return you a list of options to choose from.
+??? info We have used the openipmi tool so far to query the IPMI interface Zabbix however uses the OpenIPMI library for this. So a working ipmitool is not guarantee that Zabbix will work or vice versa. Also if you use the container for testing then the last command will not work as the test container is rather limited in what it can do. For more options check out the webpage of the owner https://github.com/vapor-ware/ipmi-simulator.
Let us launch the following command to see the Power status of our machine:
# ipmitool -H 192.168.0.23 -U ADMIN -I lanplus -P ADMIN chassis power status
Unable to Get Channel Cipher Suites
Chassis Power is
¶
Configure LLD with IPMI¶
Monitor discrete sensors¶
Ended: Monitoring SNMP, IPMI and JAVA
Authentication ↵
Authentication with HTTP¶
Authentication with LDAP¶
Authentication with SAML¶
Zabbix MFA support¶
We all know that before you can start configuring Zabbix via WebUI you have to sign in. Zabbix has several options to provide better security for user passwords by configuring password policy:
- Requirement for Minimum password length
- Requirements for password to contain an uppercase and a lowercase Latin letter, a digit, a special character
- Requirement to avoid easy-to-guess passwords
To secure sign in process even more you can configure multi factor authentication (MFA). MFA protects Zabbix by using a second source of validation before granting access to its WebUI after a user enters his/her password correctly. Zabbix offers to types of MFA - Time-based one-time password (TOTP) and Duo MFA provider.
Time-based one-time password¶
In the menu select Users
section and then Authentication
Now in MFA settings
tab select the Enable multi-factor authentication
check-box, then select TOTP
in Type drop-down list.
In Hash function
drop-down list you can choose SHA-1, SHA-256 or SHA-512, the higher number is the better security.
In Code lentgh
you can select how many digits will be generated for you by Authenticator application on your phone.
Click Add
and then Update
. Now you have TOTP MFA configured and it is the default method of MFA.
Now you need to tell Zabbix for which User group (or groups) to use MFA. Let's create a User group that would require MFA.
In the menu select Users
section and then User groups
, then click Create user group
button
In Group name
put "test". Note that Multi-factor authentication
field is "Default", as currently we have only one MFA method configured it does not matter whether we select "Default" or "TOTP1" that we created above. You also can disable MFA for all users belonging to this User group. Click Add
button to create "test" User group.
Note
MFA method is defined on per User group basis, i.e. MFA method configured for a User group will be applied to all users belonging to this group.
Let's add a user to this user group. In the menu select Users
section and then Users
, then click Create user
button
Fill in Username
, Password
and Password (once again)
fields. Make sure you select test
user group in Groups
field.
Then switch to Permissions
tab and select any role.
Click Add
button to add the user.
Now we can test how TOTP MFA works. Sign out and then try to sign in as a test user you just created. You will be presented with a QR code. That means that the user test
has not been enrolled in TOTP MFA yet.
On your phone you need to install either "Microsoft authenticator" or "Google authenticator" application. The procedure of adding new QR code is quite similar, here is how it looks in "Google authenticator". Tap Add a code
and then Scan a QR code
. You'll be immediately presented with a 6 digit code (remember we selected 6 in Code length
when we configured TOTP MFA?)
Enter this code into Verification code
field of your login screen and click Sign in
, if you did everything right you are logged in into Zabbix at this point. At this point the user "test" is considered enrolled into TOTP MFA and Zabbix stores a special code used for furthe authentications in its database. The next time user "test" tries to login into Zabbix there will be only a field to enter verification code
Warning
For TOTP MFA to work your Zabbix server must have correct time. Sometimes it's not the case especially if you are working with containers so pay attention to this.
If a user changes (or loses) his/her phone, then Zabbix administrator should reset his/her enrollment. To do that in the menue select Users
then mark a check-box to the left of "test" user and click "Reset TOTP secret" button.
After you reset TOTP secret the "test" user will have to undergo enrollment procedure again.
Duo MFA provider¶
Duo is a very famous security platform that provides a lot of security related features/products. To read more please visit https://duo.com/ . Here we'll talk about Duo only in regards to Zabbix MFA.
Warning
For Duo MFA to work your Zabbix WebUI must be configured to work with HTTPS (valid certificate is not required, self-signed certificate will work).
First of all you need to create an account with Duo (it's free to manage up to 10 users) then login into Duo, you are an admin here. In the menu on the left select Applications
and click Protect an Application
button.
Then you will see WebSDK in applications list, click on it
Here you'll see all the data needed for Zabbix.
Now let's go to Zabbix. First we need to configure Duo MFA metod. In the menue select Users
and click Authentication
. Then on MFA settings
tab click Add
in Methods
section.
Fill in all the fields with data from Duo Dashboard -> Applications -> Web SDK page (see screenshot above) and click Add
, then click Update
to update Authentication settings.
After the MFA method is configured let's switch the "Test" group to use Duo MFA. In the menu select Users
and click User groups
, then click "test" group. In the field Multi-factor authentication
select "DUO1" and click Update
.
Everything is ready. Let's test it. Sign out of Zabbix and sign back in with "test" user. You should see a welcome screen from Duo. Click several Next
buttons.
Then you need to select the method of authentication.
It is up to you what to select you can experiment with all these methods. Let's select "Duo Mobile" (you need to install "Duo mobile" application on your device). Click I have a tablet
(it's just easier to activate your device this way) and confirm that you installed "Duo mobile" on your phone. At this point you should see a QR code that you need to scan in "Duo mobile" application.
Open "Duo mobile" on your phone. If you did not have this application previously installed (thus no accounts enrolled) you will see couple of welcome screens.
Tap on "Use a QR code" and then scan the code presented by Duo in your Zabbix logiin screen. After you do that you will see that the account is enrolled to your Duo MFA. Enter account name and tap "Done" and you will see the account in the list of all account enrolled into Duo MFA on this device. In Zabbix WebUI you will also see a conirmation, click "Continue".
Duo will ask you now whether you want to add another method of authentication, click Skip for now
and you'll see a confirmation that set up completed. Click Login with Duo
and push notification will be pushed to your device.
Now just tap on "Approve" on your device and you will be logged in into Zabbix.
Duo MFA enrollment complete. If you sign out and sign in back then immediately a push notification will be sent to your device and all you need is tap on "Approve". Also you will see the user "test" in Duo where you can delete the user, or deactivate just click on it and experiment.
Ended: Authentication
Monitoring with Proxies ↵
Installing Proxies¶
There are some cases where monitoring with the Zabbix server and agents …. is not enough. Imagine monitoring external clients, different VLANs or just monitoring a DMZ where you don’t want to open for every agent a port in the firewall for the agent, SNMP, IPMI, … Here Zabbix proxies come to the rescue. A Zabbix proxy is a small remote Zabbix server that supports the collection of data. With Zabbix 7 it’s now even possible to run Zabbix proxies in HA. Proxies will also become important in larger environments to simplify the management and most importantly to offload the Zabbix server. Our proxies can do pre-processing and it will also limit the number of connections from devices to the Zabbix server.
So in short a Zabbix proxy can be used to : - Monitor remote locations - Monitor locations that have unreliable connections - Offload the Zabbix server when monitoring thousands of devices - Simplify the maintenance and management
Proxy requirements¶
If you like to setup a few proxies for test or in your environment you will need a few Linux hosts to install the Proxies on. Proxies are also available in containers so a full VM is not needed. However here we will use a VM so we can show you how to install a proxy. When it comes to proxies they are very lightweight however since Zabbix 4.2 Proxies are able to do Item value preprocessing and this can use a lot of CPU power. So the number of CPUs and memory will depends on how many machines you will monitor and how many preprocessing rules you have on your hosts.
Note
Imagine that you need to restart your Zabbix server and that all proxies start to push the data they have gathered during the downtime of the Zabbix server. This would create ahuge amount of data being sent at once to the Zabbix server and bring it to its knees in no time. Since Zabbix 6 Zabbix has added protection for overload. When Zabbix server history cache is full the history cache write access is being throttled. Zabbix server will stop accepting data from proxies when history cache usage reaches 80%. Instead those proxies will be put on a throttling list. This will continue until the cache usage falls down to 60%. Now server will start accepting data from proxies one by one, defined by the throttling list. This means the first proxy that attempted to upload data during the throttling period will be served first and until it's done the server will not accept data from other proxies.
History write cache usage | Zabbix server mode | Zabbix server action |
---|---|---|
Reaches 80% | Wait | Stops accepting proxy data, but maintains a throttling list (prioritized list of proxies to be contacted later). |
Drops to 60% | Throttled | Starts processing throttling list, but still not accepting proxy data. |
Drops to 20% | Normal | Drops the throttling list and starts accepting proxy data normally. |
Configuration¶
Zabbix actually has 2 proxy modes it supports. A Zabbix proxy can be Active or Passive just like the Zabbix agent with this exception that the agent can be both on the same time. So our first problem will be choose the Zabbix proxy that fits us. When we talk about active / passive we always talk about the proxy being active or passive. Active means the Proxy will initiate the connection to the Zabbix server, passive means it will be the Zabbix server taking contact with the proxy to gather the data.
Active vs Passive proxy¶
Zabbix proxies have been available since Zabbix 1.6, at that time they where available only as what we know today as Active proxies
. Active means that the proxy would initiate the connection by itself to the Zabbix Server.
Since version 1.8.3 passive proxies where introduced. This allowed the server to connect to the proxy. As mentioned before Zabbix agents can be both active and passive however proxies cannot be both so whe have to choose the way of the communication when we install a proxy.
Just remember that choosing the proxy mode active
or passive
has no impact on how Zabbix agents can communicate with our proxy. It's perfectly fine to have an active proxy
and a passive agent
working together.
Active proxy¶
Let's first start with the setup of an active Proxy. Things should be very simple just make sure you have a VM that you can use to install a proxy.
Zabbix Gui config¶
First we will have to add the config in our Zabbix server frontend. From our Administration menu choose the menu Proxies
.
Go to the upper right corner of the screen and press Create proxy
You will now get a pop-up like shown here. Fill in the proxy name. I have chosen for ProxyA
as it will be our active proxy.
In case you like to configure the Passive proxy use the name ProxyP
.
You can use for yourself whatever you like. In production I would suggest to use names that make it easy for you to see it's a proxy and where the proxy belongs to. Ex: what vlan or what division ....
Choose Proxy mode Active
and fill in the IP of your virtualmachine if you like to install an active proxy.
or
Choose Proxy mode Passive
and fill in the IP of your virtualmachine if you like to install a passive proxy.
When done press Add
at the bottom.
Active proxy¶
Let's first start with the setup of an active Proxy. Things should be very simple just make sure you have a VM that you can use to install a proxy.
Zabbix Gui config¶
First we will have to add the config in our Zabbix server frontend. From our Administration menu choose the menu Proxies
.
Go to the upper right corner of the screen and press Create proxy
You will now get a pop-up like shown here. Fill in the proxy name. I have chosen for ProxyA as it will be our active proxy. You can use for yourself whatever you like. In production I would suggest to use names that make it easy for you to see it's a proxy and where the proxy belongs to. Ex: what vlan or what division ....
Choose Proxy mode Active
and fill in the IP of your virtualmachine.
When done press Add
at the bottom.
Zabbix Proxy config¶
Ok now that this is done lets go to the VM where we like to install our proxy and add the Zabbix repository so we can install our needed packages.
rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rhel/9/x86_64/zabbix-release-7.0-1.el9.noarch.rpm
Zabbix provides the proxy with the option to connecto to a SQLiteDB this can be enough for most setups. In case you need more performance from the DB there is also the option to install the Proxy with a PostgreSQL or MySQL DB. We will cover how to do with the SQLite db as this is the most easy way. We show you later what to look for if installing a proxy With MySQL or PostgreSQL.
dnf install zabbix-proxy-sqlite3 zabbix-selinux-policy -y
Once this is done let's change the proxy configuration by editing the proxy configuration file.
vi /etc/zabbix/zabbix_proxy.conf
Here a few parameters needs to be changed before we can use our proxy
- Server=
- DBName=
- Hostname=
Change them according to your settings:
- Server=< zabbix server ip >
- Hostname=ProxyA ( or the name you have used in your Zabbix frontend for the Proxy )
- DBName=/tmp/zabbix_proxy.db
The parameters are explained as follows:
- Server: Same as with active agents this parameter is used to connect to the Zabbix server, we can use the IP or DNS name here.
- DBName: This depends a bit, it is usually the name of the database like we configured on the Zabbix server in case we use PostgreSQL or MySQL. However with SQLite we need to specify dbname and location. Username and Password will be ignored when using SQLite. Just make sure SELinux is properly configured else the DB will not be created. It can help to put SELinux in permissive first.
- Hostname: This parameter is used by the proxy to identify itself to the Zabbix server. This needs to be the same name we have registered in the Zabbix frontend.
Tip
Another parameter that can be useful is ListenPort this is usually 10051 for Active and Passive proxies. But if you run for some reason the proxy on the zabbix server then you have to change this port to something else.
Note
With Zabbix 7 a new parameter has been introduced on the proxy side ProxyBufferMode
. This is standard set to Hybrid
and is what is recommended. With Hybrid the proxy will buffer will work in memory and fallback to the database if needed.
The documentation states that the proxy buffer normally works like in memory mode until it runs out of memory or the oldest record exceeds the configured age. If that happens the buffer is flushed
to database and it works like in disk mode until all data have been uploaded and it starts working with memory again. On shutdown the memory buffer is flushed to database.
Note
Don't worry about the file for the db not existing Zabbix will create it by itself.
Warning
If you use DNS names make sure there is DNS caching configured on the machine. If not Zabbix will do a lookup on the DNS server everytime it wants to make a connection.
So now that we have configured everything let's start our proxy. First disable SELinux as it will block a few things this can be done by running setenforce 0
. This is only temporary and either you keep it disabled permanent or you fix the config.
An easy way is to use sealert -a /var/log/audit/audit.log
.
Once this is done there is only 1 thing that rests us to do and that is start and enable the zabbix-proxy service.
systemctl enable zabbix-proxy --now
If you look now in the frontend of the Zabbix server you will see that the proxy we have configured is Online
Note
If you like to install the proxy with MySQL or PostgreSQL as database then it is important to also install the package zabbix-sql-scripts
. There is a special shema to be used for the proxy database that can be found under /usr/share/zabbix-sql-scripts/mysql|postgresql>/proxy.sql
.
Also in this case don't forget to edit the zabbix_proxy.conf file and add DBHost, DBUser, DBPassword and DBSchema as needed.
Passive proxy¶
Lets try to setup our passive proxy
.
Things should be very simple just make sure you have a VM that you can use to install a proxy.
Zabbix Gui config¶
First we will have to add the config in our Zabbix server frontend. From our Administration menu choose the menu Proxies
.
Go to the upper right corner of the screen and press Create proxy
You will now get a pop-up like shown here. Fill in the proxy name. I have chosen for ProxyP as it will be our passive proxy. You can use for yourself whatever you like.
In production I would suggest to use names that make it easy for you to see it's a proxy and where the proxy belongs to. Ex: what vlan or what division ....
Choose Proxy mode Passive
and fill in the IP of your virtualmachine. Here besides IP we also have the option to specify a port. This is standard 10051, Zabbix proxy will listen on the same port as our Zabbix server does.
When done press Add
at the bottom.
Zabbix Proxy config¶
Ok now that this is done lets go to the VM where we like to install our proxy and add the Zabbix repository so we can install our needed packages.
rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rhel/9/x86_64/zabbix-release-7.0-1.el9.noarch.rpm
Zabbix provides the proxy with the option to connecto to a SQLiteDB this can be enough for most setups. In case you need more performance from the DB there is also the option to install the Proxy with a PostgreSQL or MySQL DB. We will cover how to do with the SQLite db as this is the most easy way. We show you later what to look for if installing a proxy With MySQL or PostgreSQL.
dnf install zabbix-proxy-sqlite3 zabbix-selinux-policy -y
Once this is done let's change the proxy configuration by editing the proxy configuration file.
vi /etc/zabbix/zabbix_proxy.conf
Here a few parameters needs to be changed before we can use our proxy
- ProxyMode=
- Server=
- DBName=
- Hostname=
Change them according to your settings:
- ProxyMode=1
- Server=< zabbix server ip >
- Hostname=ProxyA ( or the name you have used in your Zabbix frontend for the Proxy )
- DBName=/tmp/zabbix_proxy.db
The parameters are explained as follows:
- ProxyMode: This is standard 0 for Active proxy we have to change this to 1 this will turn our proxy into a Passive proxy. Be careful there is a
#
in front of the parameter this needs to be removed. If not Zabbix will not read this value in stay in Active mode. - Server: Same as with passive agents. We have to specify the IP or DNS name or a list of IPs from zabbix servers that are allowed to connect to this proxy.
- DBName: This depends a bit, it is usually the name of the database like we configured on the Zabbix server in case we use PostgreSQL or MySQL. However with SQLite we need to specify dbname and location. Username and Password will be ignored when using SQLite. Just make sure SELinux is properly configured else the DB will not be created. It can help to put SELinux in permissive first.
- Hostname: This parameter is used by the proxy to identify itself to the Zabbix server. This needs to be the same name we have registered in the Zabbix frontend.
Tip
Another parameter that can be useful is ListenPort this is usually 10051 for Active and Passive proxies. But if you run for some reason the proxy on the zabbix server then you have to change this port to something else.
Note
With Zabbix 7 a new parameter has been introduced on the proxy side ProxyBufferMode
. This is standard set to Hybrid
and is what is recommended. With Hybrid the proxy will buffer will work in memory and fallback to the database if needed.
The documentation states that the proxy buffer normally works like in memory mode until it runs out of memory or the oldest record exceeds the configured age. If that happens the buffer is flushed
to database and it works like in disk mode until all data have been uploaded and it starts working with memory again. On shutdown the memory buffer is flushed to database.
Note
Don't worry about the file for the db not existing Zabbix will create it by itself.
Warning
If you use DNS names make sure there is DNS caching configured on the machine. If not Zabbix will do a lookup on the DNS server everytime it wants to make a connection.
So now that we have configured everything let's start our proxy. First disable SELinux as it will block a few things this can be done by running setenforce 0
. This is only temporary and either you keep it disabled permanent or you fix the config.
An easy way is to use sealert -a /var/log/audit/audit.log
.
Once this is done there is only 1 thing that rests us to do and that is start and enable the zabbix-proxy service.
systemctl enable zabbix-proxy --now
If you look now in the frontend of the Zabbix server you will see that the proxy we have configured is Offline.
This is because we have configured a Passive agent remember we had to specify the IP and the Port in our Zabbix frontend? So we need to open port 10051/tcp for our zabbix server.
firewall-cmd --new-zone=proxy-access --permanent
firewall-cmd --zone=proxy-access --add-source=<zabbix server ip> --permanent
firewall-cmd --zone=proxy-access --add-port=10051/tcp --permanent
firewall-cmd --reload
We can now check our access rules in the firewall:
# firewall-cmd --zone=proxy-access --list-all
proxy-access (active)
target: default
icmp-block-inversion: no
interfaces:
sources: xxx.xxx.xxx.xxx
services:
ports: 10051/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
When we now look at our frontend our proxy should be online
Note
If you like to install the proxy with MySQL or PostgreSQL as database then it is important to also install the package zabbix-sql-scripts
. There is a special shema to be used for the proxy database that can be found under /usr/share/zabbix-sql-scripts/mysql|postgresql>/proxy.sql
.
Also in this case don't forget to edit the zabbix_proxy.conf file and add DBHost, DBUser, DBPassword and DBSchema as needed.
Proxy loadbalancing¶
With the introduction of Zabbix 7.0 proxy loadbalancing has been introduced. After running the server in a native HA setup this was probably the next logical step to add. So like the name implies it's not just a simple HA solution but Zabbix proxies will loadbalance the hosts connected to them in an automated way.
Zabbix has added something new to manage how we group those proxies together with the obvious name Proxy groups
and it works for active and for passive proxies.
Zabbix is able to detect 3 states in our group with proxies
- Online: This implies that everything is working as expected so there was communication with all the proxies.
- Offline: The minimum number of proxies defined in our proxy config group has not been met.
- Recovering: Some proxies are offline but we still have enough proxies online to not drop below the setting
minimum proxies
.
Configuration¶
The first thing we have to do is of course set up a couple of proxies. Here I will make use of 3 active proxies but feel free to use passives if you prefer or to do a mix. If you don't know how to install a proxy take a look at the installation guides for Active proxies or Passive proxies.
I have set up 3 proxies for this example:
- ProxyA1
- ProxyA2
- ProxyA3
each of them has a unique IP and hostname.
So once you have set up 3 proxies and have added them into the frontend like we did for active and passive proxies you should have a similar looking config in your Administration frontend under Proxies.
Before we start to create a loadbalancing group there are a few steps we need to fix first. We need to monitor our proxies and we also need to install some Zabbix agents on the proxies. To install the agents we have to follow a few simple steps.
On every proxy run the following commands:
dnf install zabbix-agent
We already installed the zabbix repository when we installed the proxy so no need to add it again. Now we have to adjust a few configuration options.
- Server: A list of IP adressess or DNS names pointing to our zabbix servers. This is for the Passive agent only
- ServerActive: A list of IP adressess or DNS names pointing to our zabbix servers. This is for the Active agent only.
- Hostname: The hostname of our proxy server
So set the unique hostname and point every proxy for Server and or ServerActive to the IP of your Zabbix proxy and not the Zabbix server. Our Proxy is in this case acting as our Zabbix server.
Do this for all the agents that are installed on the proxies and when done start the agents and enable them to start automatic.
systemctl enable zabbix-agent --now
Our next step will be to create our proxies in the Zabbix frontend to monitor them.
For this we go to Data collection -> Hosts
and click on Create host
.
Next we have to fill in a few fields:
- Host name: Name of the agent on the proxy. I used the same name for my agent as on the proxy this makes it easy to see when a proxy is down. Ex ProxyA1, ProxyA2, ProxyA3
- Templates: We will want to monitor the OS and the proxy so we have to apply 2 templates : Linux by Zabbix agent and Zabbix proxy health
- Host groups: Here we choose a group to put our proxies is you can choose what you want this is just a logical grouping used for permissions.
- Interface: Every hosts needs to have an interface where we define the IP of the host and the port.
Make sure your proxies are properly monitored before you continue to the next step.
Configure proxy groups¶
As one of the final steps we have to Create a proxy group
to gather all proxies we want to use for our loadbalancing.
In our Zabbix menu go to Administration -> Proxy groups
Click in the upper right corner on Create proxy group
.
Here we have to define a few options:
- Name : The name of our proxy group. This can be anything you like I have chosen for the group
Active proxies
because I have setup 3 proxies for this test. - Failover period: Period in seconds before failover is executed (1m by default; allowed range 10s-15m).
- Minimum number of proxies: The minimum number of online proxies required for the group to be online (1 by default; allowed range 1-1000). Since we have 3 proxies I added 2 as minimum.
Next we have to configure our 3 existing proxies to join this proxy group.
Go in the Zabbix menu to Administration -> Proxies
click on every proxy in the list you would like to add to the group and enter following details.
- Proxy group:
Active proxies
(or whatever name you gave to your proxy group.) - Proxy address: The active proxy ip, If specified then active proxy requests are only accepted from this list of comma-delimited IP addresses, optionally in CIDR notation, or DNS names of active Zabbix proxy.
- Address for active agents: The address and port where active agents will connect to. Usually same IP as the IP from the proxy but not needed to be the same.
- Proxy Mode: Configure if it proxy is Active or Passive
The next step is now to reconfigure our Agents on our proxies so that they are monitored by The proxy group Active Proxies
instead of only the proxy they run on.
Go to Data collection -> Hosts
and click on every host that you have configured to monitor the proxy agent. Switch Monitored by Proxy
to Proxy group
.
Because our proxies will accept active connections make sure to open firewall port 10051 on all the proxies and because we have Agents that we monitor passive make sure port 10050 is also open
firewall-cmd --add-port=10051/tcp --permanent
firewall-cmd --add-port=10050/tcp --permanent
firewall-cmd --reload
Now there is another step that we need to fix. On our Zabbix Agents we need to add all the proxy addresses to our Zabbix agents configuration.
open the file /etc/zabbix/zabbix_agentd.conf
and add a comma delimited list of IPs for Server
and ServerActive
pointing to all your proxies
This way the agent can sent active data to the proxies or it can be polled by all 3 proxies
Note
Don't forget to restart your agents and proxies when making changes in the configuration files.
Loadbalancing and HA¶
Ok now that we have configured our proxies and agents properly and made a proxy group everything is up and running. When you go to Data Collection -> Hosts
you will see that your proxie agents are monitored by the group Active proxies
. It is possible that your ProxyA1 is monitored by ProxyA2 or A3 so don't panic if your dashboard is not the same. Zabbix proxies will besides HA also do loadbalancing.
It's time to test this so let us stop one of our proxies for example ProxyA2
systemctl stop zabbix-proxy
Since we configured our Proxy group failover period to be 1m it can take 1m before you will see changes
When you look at the log files of the agent you will see that he is not able to connect the proxy anymore and that he will be redirected to another proxy in the group
1391:20240528:115433.344 Unable to connect to [xxx.xxx.xxx.xxx]:10051 [cannot connect to [[xxx.xxx.xxx.xxx]:10051]: connection error (POLLERR,POLLHUP)]
1391:20240528:115433.344 Unable to send heartbeat message to [xxx.xxx.xxx.xxx]:10051 [cannot connect to [[xxx.xxx.xxx.xxx]:10051]: connection error (POLLERR,POLLHUP)]
1392:20240528:115433.344 Unable to connect to [xxx.xxx.xxx.xxx]:10051 [cannot connect to [[xxx.xxx.xxx.xxx]:10051]: connection error (POLLERR,POLLHUP)]
1392:20240528:115433.353 Unable to connect to [xxx.xxx.xxx.xxx]:10051 [cannot connect to [[xxx.xxx.xxx.xxx]:10051]: connection error (POLLERR,POLLHUP)]
1392:20240528:115433.355 Unable to send heartbeat message to [xxx.xxx.xxx.xxx]:10051 [sequential redirect responses detected]
1390:20240528:115433.407 Unable to connect to [xxx.xxx.xxx.xxx]:10051 [cannot connect to [[xxx.xxx.xxx.xxx]:10051]: connection error (POLLERR,POLLHUP)]
1390:20240528:115433.408 Unable to connect to [xxx.xxx.xxx.xxx]:10051 [cannot connect to [[xxx.xxx.xxx.xxx]:10051]: connection error (POLLERR,POLLHUP)]
1390:20240528:115433.408 Unable to send heartbeat message to [xxx.xxx.xxx.xxx]:10051 [sequential redirect responses detected]
1392:20240528:115533.508 Successfully sent heartbeat message to [xxx.xxx.xxx.xxx]:10051
1390:20240528:115533.562 Successfully sent heartbeat message to [xxx.xxx.xxx.xxx]:10051
1392:20240528:115558.583 Active check configuration update from [xxx.xxx.xxx.xxx:10051] is working again
1391:20240528:115558.598 Active check configuration update from [xxx.xxx.xxx.xxx:10051] is working again
Warning
There are a few limitations when using proxies in HA/Loadbalancing. - SNMP traps are not supported by proxies in proxy group. - Checks depending on external configuration must have the same configuration on all proxies in proxy group. That includes: - external checks - scripts; - database checks - odbc configuration. - When monitored in proxy group the VMware hosts will be randomly spread between proxies in the group and will cause each proxy to cache all VMware data causing additional load to vCenter.
Ended: Monitoring with Proxies
Securing Zabbix ↵
Securing Zabbix Frontend¶
The frontend is what we use to login into our system. The Zabbix frontend will connect to our Zabbix server and our database. But we also send information from our laptop to the frontend. It's important that when we enter our credentials that we can do this in a safe way. So it makes sense to make use of certificates and one way to do this is by making use of Self-Signed certificates.
To give you a better understanding of why your browser will warn you when using self signed certificates, we have to know that when we request an SSL certificate from an official Certificate Authority (CA) that you submit a Certificate Signing Reauest (CSR) to them. They in return provide you with a Signed SSL certificate. For this they make use of their root certificate and private key. Our browser comes with a copy of the root certificate (CA) from various authorities or it can access it from the OS. This is why our self signed certificates are not trusted by our browser, we don't have any CA validation. Our only workaround is to create our own root certificate and private key.
Understanding the concepts¶
How to create an SSL certificate¶
How SSL works - Client - Server flow¶
Note
Borrowed the designs from https://www.youtube.com/watch?v=WqgzYuHtnIM this video explains well how SSL works.
Securing the Frontend with Self signed SSL on Nginx¶
To configure this there are a few steps that we need to follow:
- Generate a private key for the CA ( Certificate Authority )
- Generate a root certficate
- Generating CA-Authenticated Certificates
- Generate a Certificate Signing Request (CSR)
- Generate an X509 V3 certificate extension configuration file
- Generate the certificate using our CSR, the CA private key, the CA certificate, and the config file
- Copy the SSL certificates to our Virtual Host
- Adapt your Nginx Zabbix config
Generate a private key for the CA¶
First step is to make a folder named SSL so we can create our certificates and safe them:
>- mkdir ~/ssl
>- cd ~/ssl
>- openssl ecparam -out myCA.key -name prime256v1 -genkey
Let's explain all the options;
- openssl : The tool to use the OpenSSL library, this library provides us with cryptographic functions and utilities.
- out myCA.key : This part of the command specifies the output file name for the generated private key.
- name prime256v1: Name of the elliptic curve; X9.62/SECG curve over a 256 bit prime field
- ecparam: This command is used to manipulate or generate EC parameter files.
- genkey: This option will generate a EC private key using the specified parameters.
Generate a Root Certificate¶
openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem
Let's explain all the options;
- openssl: The command-line tool for OpenSSL.
- req: This command is used for X.509 certificate signing request (CSR) management.
- x509: This option specifies that a self-signed certificate should be created.
- new: This option is used to generate a new certificate.
- nodes: This option indicates that the private key should not be encrypted. It will generates a private key without a passphrase, making it more convenient but potentially less secure.
- key myCA.key: This specifies the private key file (myCA.key) to be used in generating the certificate.
- sha256: This option specifies the hash algorithm to be used for the certificate. In this case, SHA-256 is chosen for stronger security.
- days 1825: This sets the validity period of the certificate in days. Here, it’s set to 1825 days (5 years).
- out myCA.pem: This specifies the output file name for the generated certificate. In this case, “myCA.pem.”
The information you enter is not so important but it's best to fill it in as good as possible. Just make sure you enter for CN you IP or DNS.
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:BE
State or Province Name (full name) []:vlaams-brabant
Locality Name (eg, city) [Default City]:leuven
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:192.168.0.134
Email Address []:
Generating CA-Authenticated Certificates¶
It's probably good practice to use de dns name of your webiste in the name fo the private key.
As we use in this case no DNS but an IP address I will use the fictive dns zabbix.mycompany.internal.
openssl genrsa -out zabbix.mycompany.internal.key 2048
Generate a Certificate Signing Request (CSR)¶
openssl req -new -key zabbix.mycompany.internal.key -out zabbix.mycompany.internal.csr
You will be asked the same set of questions as above. Once again your answers hold minimal significance and in our case no one will inspect the certificate so they matter even less.
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:BE
State or Province Name (full name) []:vlaams-brabant
Locality Name (eg, city) [Default City]:leuven
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:192.168.0.134
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Generate an X509 V3 certificate extension configuration file.¶
# vi zabbix.mycompany.internal.ext
Add the following lines in your certificate extension file. Replace IP or DNS with your own values.
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 192.168.0.133
#DNS.1 = MYDNS (You can use DNS if you have a dns name if you use IP then use the above line)
Generate the certificate using our CSR, the CA private key, the CA certificate, and the config file¶
openssl x509 -req -in zabbix.mycompany.internal.csr -CA myCA.pem -CAkey myCA.key \
-CAcreateserial -out zabbix.mycompany.internal.crt -days 825 -sha256 -extfile zabbix.mycompany.internal.ext
Copy the SSL certificates to our Virtual Host¶
cp zabbix.mycompany.internal.crt /etc/pki/tls/certs/.
cp zabbix.mycompany.internal.key /etc/pki/tls/private/.
Import the CA in Linux (RHEL)¶
We need to update the CA certificate’s, run the below command to update the CA certs.
cp myCA.pem /etc/pki/ca-trust/source/anchors/myCA.crt
update-ca-trust extract
Import the CA in OSX¶
- Open the macOS Keychain app.
- Navigate to File > Import Items
- Choose your private key file (i.e., myCA.pem)
- Search for the “Common Name” you provided earlier.
- Double-click on your root certificate in the list.
- Expand the Trust section.
- Modify the “When using this certificate:” dropdown to “Always Trust”.
- Close the certificate window.
Import the CA in Windows¶
- Open the “Microsoft Management Console” by pressing Windows + R, typing mmc, and clicking Open.
- Navigate to File > Add/Remove Snap-in.
- Select Certificates and click Add.
- Choose Computer Account and proceed by clicking Next.
- Select Local Computer and click Finish.
- Click OK to return to the MMC window.
- Expand the view by double-clicking Certificates (local computer).
- Right-click on Certificates under “Object Type” in the middle column, select All Tasks, and then Import.
- Click Next, followed by Browse. Change the certificate extension dropdown next to the filename field to All Files (.) and locate the myCA.pem file.
- Click Open, then Next.
- Choose “Place all certificates in the following store.” with “Trusted Root Certification Authorities store” as the default. Proceed by clicking Next, then Finish, to finalize the wizard.
- If all went well you should find your certficate under Trusted Root Certification Authorities > Certificates
Warning
You also need to import the myCA.crt file in your OS we are not an official CA so we have to import it in our OS and tell it to trust this Certificate. This action depends on the OS you use.
As you are using OpenSSL, you should also create a strong Diffie-Hellman group, which is used in negotiating Perfect Forward Secrecy with clients. You can do this by typing:
openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048
Adapt your Nginx Zabbix config¶
Add the following lines to your Nginx configuration, modifying the file paths as needed.
Replace the the already existing lines with port 80 with this configuration. This will enable SSL and HTTP2.
# vi /etc/nginx/conf.d/zabbix.conf
server {
listen 443 http2 ssl;
listen [::]:443 http2 ssl;
server_name <ip qddress>;
ssl_certificate /etc/ssl/certs/zabbix.mycompany.internal.crt;
ssl_certificate_key /etc/pki/tls/private/zabbix.mycompany.internal.key;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
To redirect traffic from port 80 to 443 we can add the following lines above our https block:
server {
listen 80;
server_name _; #dns or ip is also possible
return 301 https://$host$request_uri;
}
Restart all services and allow https traffic¶
systemctl restart php-fpm.service
systemctl restart nginx
firewall-cmd --add-service=https --permanent
firewall-cmd --reload
When we go to our url http://<IP or DNS>/
we get redirected to our https://
page and when we check we can see that our site is secure:
Note
- To be even more secure have a loot at https://cipherlist.eu/ this page maintains a list of strong ciphers that you can use so secure your Nginx even more.
- You can test your nginx config with 'nginx -t' before you restart.
- For HTTP/2 to work you need atleast nginx 1.9.5 or later
Securing the Frontend with Let's Encrypt on Nginx¶
Securing Zabbix with SELinux¶
Ended: Securing Zabbix
Maintaining Zabbix ↵
Maintaining Zabbix¶
Ended: Maintaining Zabbix
Monitoring Windows ↵
Monitoring Windows¶
Ended: Monitoring Windows
Zabbix API ↵
Zabbix API¶
Ended: Zabbix API
Zabbix Extras ↵
Modbus monitoring with Zabbix.¶
Introduction¶
MODBUS TCP is a technology that takes MODBUS RTU packets and encapsulates this in a TCP/IP wrapper for communication over a network instead of typical RS485 serial connections. MODBUS as a technology has been around since 1979 and is one of the de-facto industrial application protocols commonly found in HVAC controllers, Generator controllers and a multitude of other PLC devices. Zabbix has the ability to monitor MODBUS using a Zabbix agent. If the Zabbix agent is running on a machine with a serial connection to the Modbus device, this monitoring can be done over serial communications. If the Modbus device is connected to a TCP/IP network, the Zabbix agent will still be required for sending/receiving commands. TCP port 502 is commonly used for modbus TCP communications.
In this example, all generator PLCs are connected to a TCP/IP network.
Due to the nature of the network used for this example, a Zabbix proxy is also required. In this case the Zabbix proxy server is a docker container on a virtual machine that can reach both the Zabbix server network and the network the PLC devices are connected too. The Zabbix agent that is needed for the Modbus commands is also running as a docker container on the same virtual machine as the Zabbix proxy server. Different networks may require different solutions. The common element for all networks for Modbus monitoring is there needs to be a Zabbix agent that can interface with the Modbus devices.
In this screenshot is the initial template creation step. The template group used in this screen shot is templates/power as this example will be using a generator PLC. Templates should be named in a self explanatory way. If monitoring a Horner XL4 PLC, it would be best to name the template “Horner XL4”
Before adding in the template, add in a default Macro under the tab macros wich is called {$DEVICE_CONNECTION} and set the value to “tcp://127.0.0.1:502”. Later when adding in hosts, each host that is added in will overwrite this value with the actual value of the connection string for that particular host. This will allow for troubleshooting in the future if communications are not established with a host.
Template Item creation¶
For a detailed explanation of all the modbus.get parameters see the Zabbix documentation:
https://www.zabbix.com/documentation/7.0/en/manual/appendix/items/modbus
The item key for modbus.get looks like this:
modbus.get[<endpoint>,<slaveid>,<function>,<address>,<count>,<type>,<endianness>,<offset>]
For creating the template of the item, in this example the following values will be set:
- Endpoint – the protocol/connection string in this example this is set to {$DEVICE_CONNECTION}
- Slaveid – for Serial communications, multiple Modbus can be on the same serial connection and this value allows addressing of those devices. For this TCP example, this value will be 1
- Function – 1 - Read Coil, 2 - Read Discrete Input, 3 - Read Holding Registers, 4 - Read Input Register – please read Modbus documentation on these 4 different modes.
- Address – this is the internal address of the registry to be read. In this example, address 42 is used. This will require in-depth knowledge of the Modbus device to properly determine.
- Count – if the function is 1 or 2, then count = 1 bit, for all other cases, (count*type)/2 = real count of registers for reading. To simplify, this is the number of registers to read with this command.
- Type – this is the data type to be returned from the Modbus device, for function 1 and 2, this should be set to bit. For function 3 and 4 there is a wide variety of options from bit to double. Please see the Zabbix documentation linked above, also you will need to fully understand your Modbus device and know what data types it returns.
- Endianness – does the Modbus device return big endian, little endian, Mid-big endian, or Mid-little endian values? For bit values that are returned the value be for big endian is used here. Again knowledge of the data type returned by the Modbus device is important
- Offset – for some Modbus devices that do not support random read access an offset has to be given. This allows the Modbus device to return a data byte, and the correct bit to be selected by Zabbix. The example below allows random read access so Offset is set to 0
Example Item key:¶
In this example the Modbus device is a generator controller. For this generator controller I have address 42 for a generator AC Fail alarm. This is a coil read operation. The Modbus.get item key will look like this:
modbus.get[{$DEVICE_CONNECTION},1,1,42,1,bit,be,0]
- {$DEVICE_CONNECTION} – this allows the use of a macro set in the host creation step
- 1 – Slaveid, for this example, MODBUS TCP is used so this is ignored
- 1 – Function, for this example, it is a read coil operation
- 42 – this is the internal address in the Modbus device to be read
- 1 – Count is set to 1 because this value is a bit for this example
- bit – The type of data for the example
- be – endianness – big endianness for bit operations
- 0 – the offset for this example.
Full Item template example¶
This is the full example of the item creation window. - Name – Generator AC fail – this is the name of the item we are creating - Type – Zabbix agent – This is used so that the Modbus.get command can be used - Key – modbus.get – explained above for this example - Type of information – In this case the value is either 0 or 1 so we used numeric (unsigned) - Units – There are no units to assign to this item - Update interval – 15m, this could be quicker if the Modbus device has data at a quicker interval, for this example 15m was used as a proof of concept - Value Mapping – A map was created called Alarm status where 0 = OK and 1 = Alarm which matches the expected data from this Modbus device
Host Creation¶
After creating the template, with all the items to be monitored, the hosts can be created and assigned to the template In this example:
- Host name – Test PLC for this example host
- Templates – The generic PLC template from the above example was used
- Host Group – Since this will eventually be used for a generator Modbus PLC I have a generator/Modbus host group created
- Interfaces – We use a Zabbix Agent interface with a DNS name of the Zabbix agent running on the same virtual machine as the Zabbix Proxy. This is setup due to the network configuration presented above. The Zabbix agent and Zabbix proxy are docker containers on the virtual machine with connections to both the PLC network and the Zabbix server. To communicate with a Modbus device a zabbix agent needs to be used.
- Monitored by proxy – The proxy server that will communicate with the Zabbix agent
Host Creation – Macro definition¶
Before adding the host in, the device connection string needs to be applied. In this case we change {$DEVICE_CONNECTION} to the actual value we will be using for this Modbus device. Doing it this way will allow for the template to be used over and over with different Modbus devices.
Network Diagram explained¶
- The Zabbix Server requests the Zabbix proxy to get the item value
- The Zabbix proxy requests the modbus.get command from the Zabbix agent
- The Zabbix Agent executes the modbus.get command to the PLC via Modbus TCP in this example
- The PLC responds to the Modbus TCP command to the Zabbix Agent
- The Zabbix agent returns the results of the modbus.get query to the Zabbix proxy
- The Zabbix proxy returns the item value to the zabbix server.
By : Jason Drummond
Contributors¶
Thanks to all the contributors to this book for helping me with translations and articles. Let me know if you don't see your name and have contributed to this book.
- Bgmot - https://bgmot.com/
- Jason Drummond - https://www.linkedin.com/in/jason-drummond/
- David Moreau - https://github.com/david-moreau
Ended: Zabbix Extras
Guidelines For Translations¶
How to tranlate¶
- Index pages are in the docs folder the english version is
index.md
if we like to tranlate to another language we need to name the pageindex.<language code>.md
. For example index.fr.md, the correct codes are in mkdocs.md The book images have to be created in a folder in mkdocs with the language code Ex: nl, fr - Translation of the menu can be done in mkdocs.yml. Create the option nav_translations: and translate the menu structure. Example:
- locale: nl
name: Nederlands
build: true
nav_translations:
Welcome: Welkom
Getting started: Aan de slag
The basics: De basis
Problem detection: Probleem detectie
- If you like to translate a page then create a file with the same name as the original but add the country code before .md Ex : Requirements.md will become Requirements.nl.md. If you don't make screenshots with translations then use the same images in the image dir.
- If you like to add new screenshots create a new folder image.
Ex: image/ becomes image.nl/
How to write a topic¶
- All is written in markdown , HTML is supported but try to avoid if not needed
- Start the page with the Header #
- When you write a topic start with an introduction what the topic is about
- Give an overview of the topics that you will cover
- Add visuals if possible this explains better to people how things work
- Create a how to so that people see how they can do this
- place the images in a /image folder
- You can use the english screenshots in the image folder or create your own screenshots in local language use /image.
ex: /image.fr
- You can use the english screenshots in the image folder or create your own screenshots in local language use /image.
- Close every topic with --- this will draw a horizontal line
Translation Table¶
- This table can be used to track what has been finished and is ready for translation. For this we mark the page with .
- Once other languages are translated we mark them with
- In case a page gets updated with new information after pages are already translated we mark the page with and remove the from the tranlated pages.
- Webpage is ready for translation:
- Webpage is translated :
- Webpage has been updated after translation:
- Webpage is still not finished :
Note
Please don't update this table this is for reference only when doing translations. I will maintain this table when I add new topics or merge changes.
Webpage | English | French | German | Italian | Dutch | Portugese | Spanish | Thai | Chinese |
---|---|---|---|---|---|---|---|---|---|
Welcome | |||||||||
Getting started | |||||||||
Requirements | |||||||||
Installing Zabbix DB Server | |||||||||
Installing Zabbix | |||||||||
Configure Zabbix HA | |||||||||
The basics | |||||||||
Zabbix Interface | |||||||||
Zabbix Users & User groups | |||||||||
Zabbix hosts | |||||||||
Host groups | |||||||||
Interfaces | |||||||||
templates | |||||||||
Items | |||||||||
Zabbix triggers | |||||||||
Macros | |||||||||
Data Flow | |||||||||
Data collection | |||||||||
Zabbix Agent | |||||||||
Problem detection | |||||||||
Triggers | |||||||||
Taking action when problems come | |||||||||
Event based Actions | |||||||||
Managing permissions | |||||||||
Managing Permissions | |||||||||
Visualising Problems | |||||||||
Visualising our problems | |||||||||
Automating configuration | |||||||||
Automating configuration | |||||||||
VMWare monitoring | |||||||||
VMware monitoring with Zabbix | |||||||||
Monitoring websites | |||||||||
Monitoring websites | |||||||||
Monitoring SNMP, IPMI and JAVA | |||||||||
SNMP Monitoring | |||||||||
SNMP trap monitoring | |||||||||
JAVA monitoring | |||||||||
IPMI Monitoring | |||||||||
Authentication | |||||||||
Authentication with HTTP | |||||||||
Authentication with LDAP | |||||||||
Authentication with SAML | |||||||||
Zabbix MFA support | |||||||||
Monitoring with Proxies | |||||||||
Installing Proxies | |||||||||
Active proxy | |||||||||
Passive proxy | |||||||||
Proxy loadbalancing | |||||||||
Securing Zabbix | |||||||||
Securing Zabbix Frontend | |||||||||
Securing Zabbix with SELinux | |||||||||
Maintaining Zabbix | |||||||||
Maintaining Zabbix | |||||||||
Monitoring Windows | |||||||||
Monitoring Windows | |||||||||
Zabbix API | |||||||||
Zabbix API | |||||||||
Zabbix extras | |||||||||
Modbus monitoring with Zabbix | |||||||||