Saltstack : state for monitorix + ASD to monitor docker (Archlinux)
As I’m using monitorix to monitor all my servers, I naturally did a dedicated state for saltstack. I also coupled this with a state to install Anything-sync-Daemon which is an Archlinux’s AUR Package that use tmpfs together with overlayfs to reduce wear on physical disk (data are stored to memory and synchronized on a regular basis on disk).
I also use states to add specific monitoring for some services like docker.
Build scripts
Monitorix and Anything-sync-daemon are both available from AUR on Archlinux. I just use some small scripts to quickly build the binary packages without any fingerprint on the host used to build. The two scripts a very similar and are just used to save few lines of shell commands ( yes, I’m that lazy…)
Anything-sync-daemon
build-asd.sh :
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
#!/bin/bash WORKDIR="asd" mkdir $WORKDIR cd $WORKDIR echo "" echo "------ get sources package... -------" git clone https://aur.archlinux.org/anything-sync-daemon.git echo "" echo "------ building... -------" cd anything-sync-daemon makepkg echo "" echo "------ finalizing... -----" mv anything-sync-daemon-*-any.pkg.tar.xz ../../ cd ../../ rm -Rf asd |
Monitorix
build_monitorix.sh:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
#!/bin/bash WORKDIR="monitorix" mkdir $WORKDIR cd $WORKDIR echo "" echo "------ get sources package... -------" git clone https://aur.archlinux.org/monitorix.git echo "" echo "------ building... -------" cd monitorix makepkg echo "" echo "------ finalizing... -----" mv monitorix-*-any.pkg.tar.xz ../../ cd ../../ rm -Rf monitorix |
States
The states below basicaly handle dependencies, packages installation and custom configuration file deployment.
Anything-sync-daemon
The ASD state is pretty simple:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
# needed package for asd neededpkgs: pkg.installed: - pkgs: - procps-ng - rsync # deploy custom package /var/cache/pacman/pkg/anything-sync-daemon-5.76-1-any.pkg.tar.xz: file.managed: - source: salt://anything-sync-daemon/anything-sync-daemon-5.76-1-any.pkg.tar.xz - user: root - group: root - mode: 644 pacman --noconfirm -U /var/cache/pacman/pkg/anything-sync-daemon-5.76-1-any.pkg.tar.xz: cmd.run: - creates: /usr/bin/anything-sync-daemon # ensure service is running asd: service: - running - watch: - file: /etc/asd.conf # deploy custom config /etc/asd.conf: file: - managed - source: salt://anything-sync-daemon/asd.conf - user: root - group: root - mode: 644 /etc/modules-load.d/overlay.conf: file.managed: - source: salt://anything-sync-daemon/overlay.conf - user: root - group: root - mode: 644 |
The “overlay.conf ” is used to activate the overlay kernel module at boot.
The reason I use a custom configuration for ASD file (asd.conf) is to allow adding directories to sync. In fact, I’m using ASD both for monitorix and docker, so my custom config file contains markers to easily adapt the config file in other states (eg. “# monitorix start” and “# monitorix end”)
The pushed asd.conf file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
# # /etc/asd.conf # # For documentation, refer to the asd man page ## WARNING Do NOT edit anything in this file while asd is running! ## To protect data from corruption, in the event that you do make an edit ## while asd is active, any changes made will be applied the next time ## you start-up asd. # Define the target(s) directories in the WHATTOSYNC array # Do NOT define a file! These MUST be directories with an absolute path! # # Note that the target DIRECTORIES and all subdirs under them will be included. # In other words, this is recursive. # # Below is an example to whet your appetite. #WHATTOSYNC=('/srv/http' '/var/lib/monitorix' '/foo/bar') WHATTOSYNC=( '/var/log' # monitorix start # monitorix end # docker start # docker end ) # Define where data will reside in tmpfs. # This location must be mounted to tmpfs and MUST be writable and executable. # # If using bleachbit, do NOT invoke it with the '--clean system.tmp' switch or # you will remove a key dot file (.foo) from /tmp that asd needs to keep track # of sync status. # # Note that using a value of '/dev/shm' can cause problems with systemd's # NAMESPACE spawning only when users enable the overlayfs option. # # Use NO trailing backslash! VOLATILE="/tmp" # Uncomment and set to yes to use an overlayfs instead of a full copy to reduce # the memory costs and to improve sync/unsync operations. # # You must modprobe either the 'overlayfs' or 'overlay' module prior to running asd if # you enable this option. Distros running the linux kernel version >=3.18.0 are likely # using the 'overlay' module while some distros shipping older kernels, notably Ubuntu # provide the older version of this technology which is provided in the 'overlayfs' # module not 'overlay' module. USE_OVERLAYFS="yes" # Uncomment and set to no to completely disable the crash recovery feature of asd. # # The default is to create crash recovery backups if the system is ungracefully # powered-down due to a kernel panic, hitting the reset switch, battery going # dead, etc. Some users keep very diligent backups and don't care to have this # feature enabled. #USE_BACKUPS="yes" |
As shown in the monitorix state below, I use the following lines to add the corresponding ASD line in asd.conf file from the monitorix state:
|
1 2 3 4 5 6 |
monitorix_asd.conf: file.blockreplace: - name: /etc/asd.conf - marker_start: "# monitorix start" - marker_end: "# monitorix end" - content: "'/var/lib/monitorix' '/srv/http/monitorix'" |
Monitorix
The monitorix state install all required softwares and a custom configuration file, again, to allow customization depending on other states I have (eg: docker monitoring).
The monitorix state
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
# Docker image building template include: - cron monitorixpkgs: pkg.installed: - pkgs: - mesa-libgl - perl - perl-cgi - perl-mailtools - perl-mime-lite - perl-libwww - perl-dbi - perl-xml-simple - perl-config-simple - perl-config-general - rrdtool - perl-http-server-simple /var/cache/pacman/pkg/monitorix-3.9.0-1-any.pkg.tar.xz: file: - managed - source: salt://monitorix/monitorix-3.9.0-1-any.pkg.tar.xz - user: root - group: root - mode: 644 pacman --noconfirm -U /var/cache/pacman/pkg/monitorix-3.9.0-1-any.pkg.tar.xz: cmd.run: - creates: /usr/bin/monitorix /etc/monitorix/monitorix.conf: file.managed: - source: salt://monitorix/monitorix.conf - user: root - group: root - mode: 644 monitorix_title_conf: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# title start" - marker_end: "# title end" - content: "title = {{ grains['host'] }} monitoring" monitorix_hostname_conf: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# hostname start" - marker_end: "# hostname end" - content: "hostname = {{ grains['host'] }}" monitorix_asd.conf: file.blockreplace: - name: /etc/asd.conf - marker_start: "# monitorix start" - marker_end: "# monitorix end" - content: "'/var/lib/monitorix' '/srv/http/monitorix'" monitorix_email_address_conf: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# from email address start" - marker_end: "# from email address end" - content: "from_address = {{ grains['host'] }}@domain.ext" monitorix: service: - running - enable: True - watch: - file: /etc/monitorix/monitorix.conf |
My monitorix.conf file is based on the original one, on which I changed some parts to be able to change or add parameters from saltstack states with markers.
As an example, below are parts of the monitorix configuration file where I put markers for my “monitorix for docker” state.
The FS part
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# FS graph # ----------------------------------------------------------------------------- <fs> <list> # fs list start 0 = / # fs list stop </list> <desc> </desc> <devmap> # fs devmap start / = mmcblk0p2 # fs devmap stop </devmap> rigid = 2, 0, 2, 0 limit = 100, 1000, 100, 1000 <alerts> / = 3600, 75, /usr/local/bin/alert_diskspace_root # fs alerts start 1 # fs alerts stop 1 </alerts> </fs>< |
;
As you can see, I use a custom script for disk space alert handling (which are also deployed as a state), as explained in the official monitorix documentation
The Du part
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# DU graph # ----------------------------------------------------------------------------- <du> list = System, Users <desc> 0 = /tmp, /var/log, /var/lib/monitorix # du desk start 1 # du desk end 1 </desc> <dirmap> /var/spool/mail = Mail boxes /var/spool/mqueue = Mail queue </dirmap> graphs_per_row = 2 rigid = 0 limit = 100 </du> |
The process part
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# PROCESS graph # ----------------------------------------------------------------------------- <process> <list> # process list start 1 0 = sshd # process list end 1 </list> <desc> # process desc start 1 # process desc end 1 # process desc start 2 # process desc end 2 # process desc start 3 # process desc end 3 # process desc start 4 # process desc end 4 </desc> rigid = 0, 0, 0, 0 limit = 1000, 1000, 1000, 1000 </process> |
Monitorix state for docker monitoring
I use a “sub monitorix state” to add docker monitoring for hosts that run docker containers. This state is named “4docker.sls” and include the monitorix state. So adding monitoring support for docker host is easily done with a call like :
|
1 |
salt 'mydockerhost.local.lan' state.apply monitorix.4docker |
The 4docker.sls file :
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
include: - monitorix - custom_bin # activate disk monitoring add_docker_disk_check: file.line: - name: /etc/monitorix/monitorix.conf - mode: replace - match: disk = - content: disk = y config-4docker-fs-1: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# fs list start" - marker_end: "# fs list stop" - content: 0 = /,/var/lib/docker/volumes config-4docker-fs-2: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# fs alerts start 1" - marker_end: "# fs alerts stop 1" - content: /var/lib/docker/volumes = 7200, 80, /usr/local/bin/alert_diskspace_dockervolumes config-4docker-du-1: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# du desk start 1" - marker_end: "# du desk end 1" - content: 1 = /var/lib/docker/aufs, /var/lib/docker/image config-4docker-process-1: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# process list start 1" - marker_end: "# process list end 1" - content: 0 = sshd, docker config-4docker-process-desc-1: file.blockreplace: - name: /etc/monitorix/monitorix.conf - marker_start: "# process desc start 1" - marker_end: "# process desc end 1" - content: docker = Docker |
As a example of result, the file monitoring graph is like the following :
