System: Moving munin stats to tmpfs
The munin monitoring tool keeps track of past and present values using its own database made up of individual RRD (Round Robin Database) files for each node being tracked.
In Debian by default these files are stored under /var/lib/munin, but moving them to a tmpfs can speed things up by keeping the data in volatile memory and not writing files.
The problem with using tmpfs is that it is wiped on reboot so we need to find a way to preserve the munin state files and restore them after a reboot.
The tmpfs file system
In Debian 6 the default install includes a tmpfs file system at /dev/shm. In Debian 7 ("Wheezy") this has changed to /run/shm, but the function is identical. You can also create and mount your own tmpfs file system - see the links under References below for instructions.
We use the tmpfs file system for storing data and working files for various programs, including PostgreSQL statistics, a DNS cache, PHP sessions, duplicity state files and the mod_pagespeed cache. These are all non-essential and will be rebuilt automatically after a reboot.
In the case of munin, however, you probably don't want to lose all the graph and related data since your last reboot. You can find a solution for this below.
Moving munin files to tmpfs
The first step is to update the munin config file at /etc/munin/munin.conf. In our case that meant editing two lines to change /var/lib/munin to /dev/shm/munin (use 'run' instead of 'dev' on Wheezy).
< #dbdir /var/lib/munin
> dbdir /dev/shm/munin
< #cgitmpdir /var/lib/munin/cgi-tmp
> cgitmpdir /dev/shm/munin/cgi-tmp
These changes won't take effect until the munin-node daemon is
restarted. Running the following commands as root will copy all the
relevant files to tmpfs, preserving the ownership and permissions.
service munin-node stop
cp -rp /var/lib/munin /dev/shm/munin
service munin-node start
The munin directory and all included files should be owned by munin:munin with the exception of the cgi-tmp directory which needs to be group-writable by the webserver (munin:www-data) and its contents also owned by the webserver (www-data:www-data).
You should now see on the next invocation of munin-cron that the files under /dev/shm/munin are updating and not those under /var/lib/munin. Check also that you can still see the graphs updating. If not, reverse the configuration changes, restart the munin-node daemon and check for errors in the logfile at /var/log/munin/munin-node.log.
Preserving state after system boot
The munin state (RRD) files are updated every 5 minutes. Copying these files to the file system on every update would miss the point of using tmpfs in the first place, but using a cron script we can copy them every hour, every few hours, or every day, depending on how much data you're prepared to lose (up to the cron interval).
Here's a sample script that can be dropped into cron.hourly, cron.daily, or called via a custom crontab. For now, we're just using cron.hourly.
/etc/cron.hourly/munin-backup-graphs:
#!/bin/bash
# make backup of munin stats dir from tmpfs
BACKDIR=/var/lib/munin/
LIVEDIR=/dev/shm/munin/
if [[ -d "$LIVEDIR" && -d "$BACKDIR" ]]; then
/usr/bin/nice /usr/bin/rsync -a --delete $LIVEDIR $BACKDIR
fi
This will mirror the contents of the munin directory in tmpfs to the original location under /var/lib/munin which is no longer being used. If you want to be cautious, leave out the delete option and files will be copied as they change, but never deleted.
After a reboot the tmpfs will have been emptied, so we need to copy the contents of the backup back into tmpfs. The cleanest way to do this is to place a script in the /etc/network/if-pre-up.d directory.
/etc/network/if-pre-up.d/munin-restore-graphs:
#!/bin/bash
# restore munin stats dir to tmpfs on startup
BACKDIR=/var/lib/munin
LIVEDIR=/dev/shm/munin
if [[ -d "$BACKDIR" && ! -d "$LIVEDIR" ]]; then
/bin/cp -rp $BACKDIR $LIVEDIR
fi
This script checks that the tmpfs directory is missing and copies the backup files into place. It will run before any interfaces are brought up. Both scripts need to be executable.
Those concerned about losing even an hours worth of data can copy the backup script to the /etc/network/if-post-down.d directory to have it called during the shutdown process. In theory this would obviate the need for the cron script - except in case of a power outage.
Remember under Wheezy to replace all instances of /dev/shm with /run/shm.
Presuming that you have munin installed already you can check the "Munin processing time" graph to see the impact of moving to tmpfs. You will also see a jump in disk usage for /dev/shm.
In our case the munin directory is relatively small at 10-12Mb. For much larger systems make sure there is enough space in tmpfs and that the backup script doesn't consume too many resources.
As always, you're using these scripts at your own risk. Let us know of your experience using the Feedback form below.