File Syncronisation with Unison
What was supposed to be a quick-and-dirty on 2-way file synchronisation for Debian servers using Unison.
|2-way synchronisation||May need to build it from source code|
|Exists in Debian repositories||Not under ‘active’ development|
|Runs under Unix/Windows/OSX||More than 2 nodes Star topology recommended (ie, single point of failure)|
See Also: https://email@example.com/how-to-install-unison-91c9fb36ac09
From Debian Repos
Assuming a Debian server, we can install Unison using apt-get.
sudo apt-get update && sudo apt-get install unison
Currently (2018/09/03) this will install the following version of Unison, depending on the version of Debian you’re running:
- 8 Jessie: v2.40.102
- 9 Stretch: v2.48.3
If a later release than that published in the Debian repositories is required, we can build it from source. Grab the latest release source code from https://github.com/bcpierce00/unison/releases
Make sure we have the required OCaml compiler stuff
sudo apt-get install ocaml-native-compilers
Then build it (replacing 2.51.2 with whatever is the latest release)
wget https://github.com/bcpierce00/unison/archive/v2.51.2.tar.gz tar -zxf v2.51.2.tar.gz cd unison-2.51.2/ make all && make test && sudo HOME=/usr make install
Looks like neither of the above installation methods provide us with the unison-fsmonitor tool, which would allow us to run Unison kinda as a daemon/service with some tweaks. The unison-fsmonitor tool can watch a directory for file changes, and trigger Unison to do it’s sync-thang automatically without the need to call it manually.
sudo curl -L -o /bin/unison-fsmonitor https://github.com/TentativeConvert/Syndicator/raw/master/unison-binaries/unison-fsmonitor sudo chmod +x /bin/unison-fsmonitor
Testing it out locally
It’s possible to synchronise 2 directories on the same machine, so in the name of KISS, let’s try that out first. In this experiment we’ll try to keep the following two directories ‘in sync’:
Running Unison Manually
In our Unison ‘profile’, let’s try the following so it knows ‘sup /home/jarrod/.unison/local.prf
root = /home/jarrod/unison/local-a root = /home/jarrod/unison/local-b auto = true batch = true
Push the big red button
Apparently running Unison should be as simple as calling unison with the name of our profile (in this case ‘local’).
Well, that’s interesting.
Nothing to do: replicas have been changed only in identical ways since last sync.
Makes sense. Let’s create some test files and run it
jarrod@ha01:~/unison$ tree . |-- local-a | |-- me | |-- test-b | |-- test-c | `-- test-d `-- local-b |-- test-a |-- test-d |-- test-e |-- test-f `-- test-g 2 directories, 9 files
FUCKING YES! 😄
jarrod@ha01:~/unison$ tree . |-- local-a | |-- test-a | |-- test-b | |-- test-c | |-- test-d | |-- test-e | |-- test-f | `-- test-g `-- local-b |-- test-a |-- test-b |-- test-c |-- test-d |-- test-e |-- test-f `-- test-g 2 directories, 14 files
Remove some files, and…
jarrod@ha01:~/unison$ rm local-b/test-[a-e] jarrod@ha01:~/unison$ tree . |-- local-a | |-- test-a | |-- test-b | |-- test-c | |-- test-d | |-- test-e | |-- test-f | `-- test-g `-- local-b |-- test-f `-- test-g 2 directories, 9 files jarrod@ha01:~$ unison local jarrod@ha01:~/unison$ tree . |-- local-a | |-- test-f | `-- test-g `-- local-b |-- test-f `-- test-g 2 directories, 4 files
Ok. So it basically does what I want, but only when I tell it to. How can we automate this process?
Watch directories and sync automatically with Unison
Building on our above Unison profile, let’s use the following
root = /home/jarrod/unison/local-a root = /home/jarrod/unison/local-b auto = true batch = true repeat = watch
Now, when we run Unison with our ‘local’ profile, it’ll keep running and watch the ' root’ directories for changes.
This is great!
Where it all comes crashing down 😓
The maximum number of files that can be ‘watched’ for changes by in Linux using inotify (which I can only assume unison-fsmonitor is doing) is defined by the fs.inotify.max_user_watches system variable. By default (in Debian land, at least) this number is 8192.
# cat /proc/sys/fs/inotify/max_user_watches 8192
Ok, so usually we are able to change these values - that’s the whole point of having configuration options, right?
# echo 524288 > /proc/sys/fs/inotify/max_user_watches -su: /proc/sys/fs/inotify/max_user_watches: Permission denied
Permission fucking what?
It’s interesting to note that I’ve tried this on another “VPS” that runs under OpenVZ on my own hardware, and it appears to be a limitation of this type of virtualisation, not necessarily something that has been overridden/forbidden by a host.
Trying the same command on a KVM virtual machine is a whole ‘nother story.
# echo 8192 > /proc/sys/fs/inotify/max_user_watches # cat /proc/sys/fs/inotify/max_user_watches 8192 # echo 524288 > /proc/sys/fs/inotify/max_user_watches # cat /proc/sys/fs/inotify/max_user_watches 524288
Works like a bought one! 😉
To make the change to fs.inotify.max_user_watches permenant, ie not lost between reboots, you can add it to /etc/sysctl.conf.
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
So, this could be a viable option if A) You run your own hardware B) You aren’t running on some VM under OpenVZ
Unfortunately that’s not the situation I’m in right now 😒