Saltstack : state for monitorix + ASD to monitor docker (Archlinux)

As I’m using monitorix to monitor all my servers, I naturally did a dedicated state for saltstack. I also coupled this with a state to install Anything-sync-Daemon which is an Archlinux’s AUR Package that use tmpfs together with overlayfs to reduce wear on physical disk (data are stored to memory and synchronized on a regular basis on disk).

I also use states to add specific monitoring for some services like docker.

Build scripts

Monitorix and Anything-sync-daemon are both available from AUR on Archlinux. I just use some small scripts to quickly build the binary packages without any fingerprint on the host used to build. The two scripts a very similar and are just used to save few lines of shell commands ( yes, I’m that lazy…)

Anything-sync-daemon

build-asd.sh : 

  1. #!/bin/bash
  2.  
  3. WORKDIR="asd"
  4.  
  5. mkdir $WORKDIR
  6. cd $WORKDIR
  7.  
  8. echo ""
  9. echo "------ get sources package... -------"
  10. git clone https://aur.archlinux.org/anything-sync-daemon.git
  11.  
  12. echo ""
  13. echo "------ building... -------"
  14. cd anything-sync-daemon
  15. makepkg
  16.  
  17. echo ""
  18. echo "------ finalizing... -----"
  19. mv anything-sync-daemon-*-any.pkg.tar.xz ../../
  20. cd ../../
  21. rm -Rf asd

Monitorix

build_monitorix.sh:

  1. #!/bin/bash
  2.  
  3. WORKDIR="monitorix"
  4.  
  5. mkdir $WORKDIR
  6. cd $WORKDIR
  7.  
  8. echo ""
  9. echo "------ get sources package... -------"
  10. git clone https://aur.archlinux.org/monitorix.git
  11.  
  12. echo ""
  13. echo "------ building... -------"
  14. cd monitorix
  15. makepkg
  16.  
  17. echo ""
  18. echo "------ finalizing... -----"
  19. mv monitorix-*-any.pkg.tar.xz ../../
  20. cd ../../
  21. rm -Rf monitorix

States

The states below basicaly handle dependencies, packages installation and custom configuration file deployment.

Anything-sync-daemon

The ASD state is pretty simple:

  1. # needed package for asd
  2. neededpkgs:
  3.  pkg.installed:
  4.  - pkgs:
  5.  - procps-ng
  6.  - rsync
  7.  
  8. # deploy custom package
  9. /var/cache/pacman/pkg/anything-sync-daemon-5.76-1-any.pkg.tar.xz:
  10.  file.managed:
  11.  - source: salt://anything-sync-daemon/anything-sync-daemon-5.76-1-any.pkg.tar.xz
  12.  - user: root
  13.  - group: root
  14.  - mode: 644
  15.  
  16. pacman --noconfirm -U /var/cache/pacman/pkg/anything-sync-daemon-5.76-1-any.pkg.tar.xz:
  17.  cmd.run:
  18.  - creates: /usr/bin/anything-sync-daemon
  19.  
  20. # ensure service is running 
  21. asd:
  22.  service:
  23.  - running
  24.  - watch:
  25.  - file: /etc/asd.conf
  26.  
  27. # deploy custom config
  28. /etc/asd.conf:
  29.  file:
  30.  - managed
  31.  - source: salt://anything-sync-daemon/asd.conf
  32.  - user: root
  33.  - group: root
  34.  - mode: 644
  35.  
  36. /etc/modules-load.d/overlay.conf:
  37.  file.managed:
  38.  - source: salt://anything-sync-daemon/overlay.conf
  39.  - user: root
  40.  - group: root
  41.  - mode: 644

The “overlay.conf ” is used to activate the overlay kernel module at boot.

The reason I use a custom configuration for ASD file (asd.conf) is to allow adding directories to sync. In fact, I’m using ASD both for monitorix and docker, so my custom config file contains markers to easily adapt the config file in other states (eg. “# monitorix start” and “# monitorix end”)

The pushed asd.conf file:

  1. #
  2. # /etc/asd.conf
  3. #
  4. # For documentation, refer to the asd man page
  5.  
  6. ## WARNING Do NOT edit anything in this file while asd is running!
  7. ## To protect data from corruption, in the event that you do make an edit
  8. ## while asd is active, any changes made will be applied the next time
  9. ## you start-up asd.
  10.  
  11. # Define the target(s) directories in the WHATTOSYNC array
  12. # Do NOT define a file! These MUST be directories with an absolute path!
  13. #
  14. # Note that the target DIRECTORIES and all subdirs under them will be included.
  15. # In other words, this is recursive.
  16. #
  17. # Below is an example to whet your appetite.
  18. #WHATTOSYNC=('/srv/http' '/var/lib/monitorix' '/foo/bar')
  19. WHATTOSYNC=(
  20. '/var/log' 
  21. # monitorix start
  22. # monitorix end
  23. # docker start
  24. # docker end
  25. )
  26.  
  27. # Define where data will reside in tmpfs.
  28. # This location must be mounted to tmpfs and MUST be writable and executable.
  29. #
  30. # If using bleachbit, do NOT invoke it with the '--clean system.tmp' switch or
  31. # you will remove a key dot file (.foo) from /tmp that asd needs to keep track
  32. # of sync status.
  33. #
  34. # Note that using a value of '/dev/shm' can cause problems with systemd's
  35. # NAMESPACE spawning only when users enable the overlayfs option.
  36. #
  37. # Use NO trailing backslash!
  38. VOLATILE="/tmp"
  39.  
  40. # Uncomment and set to yes to use an overlayfs instead of a full copy to reduce
  41. # the memory costs and to improve sync/unsync operations.
  42. #
  43. # You must modprobe either the 'overlayfs' or 'overlay' module prior to running asd if
  44. # you enable this option. Distros running the linux kernel version >=3.18.0 are likely
  45. # using the 'overlay' module while some distros shipping older kernels, notably Ubuntu
  46. # provide the older version of this technology which is provided in the 'overlayfs'
  47. # module not 'overlay' module.
  48. USE_OVERLAYFS="yes"
  49.  
  50. # Uncomment and set to no to completely disable the crash recovery feature of asd.
  51. #
  52. # The default is to create crash recovery backups if the system is ungracefully
  53. # powered-down due to a kernel panic, hitting the reset switch, battery going
  54. # dead, etc. Some users keep very diligent backups and don't care to have this
  55. # feature enabled.
  56. #USE_BACKUPS="yes"

As shown in the monitorix state below, I use the following lines to add the corresponding ASD line in asd.conf file from the monitorix state:

  1. monitorix_asd.conf:
  2.  file.blockreplace:
  3.  - name: /etc/asd.conf
  4.  - marker_start: "# monitorix start"
  5.  - marker_end: "# monitorix end"
  6.  - content: "'/var/lib/monitorix' '/srv/http/monitorix'"

Monitorix

The monitorix state install all required softwares and a custom configuration file, again, to allow customization depending on other states I have (eg: docker monitoring).

The monitorix state

  1. # Docker image building template
  2. include:
  3.  - cron
  4.  
  5. monitorixpkgs:
  6.  pkg.installed:
  7.  - pkgs:
  8.  - mesa-libgl
  9.  - perl
  10.  - perl-cgi
  11.  - perl-mailtools
  12.  - perl-mime-lite
  13.  - perl-libwww
  14.  - perl-dbi
  15.  - perl-xml-simple
  16.  - perl-config-simple
  17.  - perl-config-general
  18.  - rrdtool
  19.  - perl-http-server-simple
  20.  
  21. /var/cache/pacman/pkg/monitorix-3.9.0-1-any.pkg.tar.xz:
  22.  file:
  23.  - managed
  24.  - source: salt://monitorix/monitorix-3.9.0-1-any.pkg.tar.xz
  25.  - user: root
  26.  - group: root
  27.  - mode: 644
  28.  
  29. pacman --noconfirm -U /var/cache/pacman/pkg/monitorix-3.9.0-1-any.pkg.tar.xz:
  30.  cmd.run:
  31.  - creates: /usr/bin/monitorix
  32.  
  33. /etc/monitorix/monitorix.conf:
  34.  file.managed:
  35.  - source: salt://monitorix/monitorix.conf
  36.  - user: root
  37.  - group: root
  38.  - mode: 644
  39.  
  40. monitorix_title_conf:
  41.  file.blockreplace:
  42.  - name: /etc/monitorix/monitorix.conf
  43.  - marker_start: "# title start"
  44.  - marker_end: "# title end"
  45.  - content: "title = {{ grains['host'] }} monitoring"
  46.  
  47. monitorix_hostname_conf:
  48.  file.blockreplace:
  49.  - name: /etc/monitorix/monitorix.conf
  50.  - marker_start: "# hostname start"
  51.  - marker_end: "# hostname end"
  52.  - content: "hostname = {{ grains['host'] }}"
  53.  
  54. monitorix_asd.conf:
  55.  file.blockreplace:
  56.  - name: /etc/asd.conf
  57.  - marker_start: "# monitorix start"
  58.  - marker_end: "# monitorix end"
  59.  - content: "'/var/lib/monitorix' '/srv/http/monitorix'"
  60.  
  61. monitorix_email_address_conf:
  62.  file.blockreplace:
  63.  - name: /etc/monitorix/monitorix.conf
  64.  - marker_start: "# from email address start"
  65.  - marker_end: "# from email address end"
  66.  - content: "from_address = {{ grains['host'] }}@domain.ext"
  67.  
  68. monitorix:
  69.  service:
  70.  - running
  71.  - enable: True
  72.  - watch:
  73.  - file: /etc/monitorix/monitorix.conf

My monitorix.conf file is based on the original one, on which I changed some parts to be able to change or add parameters from saltstack states with markers. 

As an example, below are parts of the monitorix configuration file where I put markers for my “monitorix for docker” state.

The FS part

  1. # FS graph
  2. # -----------------------------------------------------------------------------
  3. <fs>
  4. <list>
  5. # fs list start
  6. 0 = /
  7. # fs list stop
  8. </list>
  9. <desc>
  10. </desc>
  11. <devmap>
  12. # fs devmap start
  13. / = mmcblk0p2
  14. # fs devmap stop
  15. </devmap>
  16. rigid = 2, 0, 2, 0
  17. limit = 100, 1000, 100, 1000
  18. <alerts>
  19. / = 3600, 75, /usr/local/bin/alert_diskspace_root
  20. # fs alerts start 1
  21. # fs alerts stop 1
  22. </alerts>
  23. </fs><

;
As you can see, I use a custom script for disk space alert handling (which are also deployed as a state), as explained in the official monitorix documentation

The Du part

 

  1. # DU graph
  2. # -----------------------------------------------------------------------------
  3. <du>
  4.  list = System, Users
  5.  <desc>
  6.  0 = /tmp, /var/log, /var/lib/monitorix
  7. # du desk start 1
  8. # du desk end 1
  9.  </desc>
  10.  <dirmap>
  11.  /var/spool/mail = Mail boxes
  12.  /var/spool/mqueue = Mail queue
  13.  </dirmap>
  14.  graphs_per_row = 2
  15.  rigid = 0
  16.  limit = 100
  17. </du>

The process part

  1. # PROCESS graph
  2. # -----------------------------------------------------------------------------
  3. <process>
  4.  <list>
  5. # process list start 1
  6.  0 = sshd
  7. # process list end 1
  8.  </list>
  9.  <desc>
  10. # process desc start 1
  11. # process desc end 1
  12. # process desc start 2
  13. # process desc end 2
  14. # process desc start 3
  15. # process desc end 3
  16. # process desc start 4
  17. # process desc end 4
  18.  </desc>
  19.  rigid = 0, 0, 0, 0
  20.  limit = 1000, 1000, 1000, 1000
  21. </process>

Monitorix state for docker monitoring

I use a “sub monitorix state” to add docker monitoring for hosts that run docker containers. This state is named “4docker.sls” and include the monitorix state. So adding monitoring support for docker host is easily done with a call like :

salt 'mydockerhost.local.lan' state.apply monitorix.4docker

The 4docker.sls file :

  1. include:
  2.  - monitorix
  3.  - custom_bin
  4.  
  5. # activate disk monitoring
  6. add_docker_disk_check:
  7.  file.line:
  8.  - name: /etc/monitorix/monitorix.conf
  9.  - mode: replace
  10.  - match: disk = 
  11.  - content: disk = y
  12.  
  13. config-4docker-fs-1:
  14.  file.blockreplace:
  15.  - name: /etc/monitorix/monitorix.conf
  16.  - marker_start: "# fs list start"
  17.  - marker_end: "# fs list stop"
  18.  - content: 0 = /,/var/lib/docker/volumes
  19.  
  20. config-4docker-fs-2:
  21.  file.blockreplace:
  22.  - name: /etc/monitorix/monitorix.conf
  23.  - marker_start: "# fs alerts start 1"
  24.  - marker_end: "# fs alerts stop 1"
  25.  - content: /var/lib/docker/volumes = 7200, 80, /usr/local/bin/alert_diskspace_dockervolumes
  26.  
  27. config-4docker-du-1:
  28.  file.blockreplace:
  29.  - name: /etc/monitorix/monitorix.conf
  30.  - marker_start: "# du desk start 1"
  31.  - marker_end: "# du desk end 1"
  32.  - content: 1 = /var/lib/docker/aufs, /var/lib/docker/image
  33.  
  34. config-4docker-process-1:
  35.  file.blockreplace:
  36.  - name: /etc/monitorix/monitorix.conf
  37.  - marker_start: "# process list start 1"
  38.  - marker_end: "# process list end 1"
  39.  - content: 0 = sshd, docker
  40.  
  41. config-4docker-process-desc-1:
  42.  file.blockreplace:
  43.  - name: /etc/monitorix/monitorix.conf
  44.  - marker_start: "# process desc start 1"
  45.  - marker_end: "# process desc end 1"
  46.  - content: docker = Docker

As a example of result, the file monitoring graph is like the following :

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.