File Syncronisation with Unison

What was supposed to be a quick-and-dirty on 2-way file synchronisation for Debian servers using Unison.

Pros Cons
2-way synchronisation May need to build it from source code
Exists in Debian repositories Not under ‘active’ development
Runs under Unix/Windows/OSX More than 2 nodes Star topology recommended (ie, single point of failure)

Homepage: https://www.cis.upenn.edu/~bcpierce/unison/
GitHub: https://github.com/bcpierce00/unison
Manual: https://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html

See Also: https://medium.com/@yobi.kim/how-to-install-unison-91c9fb36ac09

Installation

From Debian Repos

Assuming a Debian server, we can install Unison using apt-get.

sudo apt-get update && sudo apt-get install unison

Currently (2018/09/03) this will install the following version of Unison, depending on the version of Debian you’re running:

From source

If a later release than that published in the Debian repositories is required, we can build it from source. Grab the latest release source code from https://github.com/bcpierce00/unison/releases

Make sure we have the required OCaml compiler stuff

sudo apt-get install ocaml-native-compilers

Then build it (replacing 2.51.2 with whatever is the latest release)

wget https://github.com/bcpierce00/unison/archive/v2.51.2.tar.gz
tar -zxf v2.51.2.tar.gz
cd unison-2.51.2/
make all && make test && sudo HOME=/usr make install

Hmm…

Looks like neither of the above installation methods provide us with the unison-fsmonitor tool, which would allow us to run Unison kinda as a daemon/service with some tweaks. The unison-fsmonitor tool can watch a directory for file changes, and trigger Unison to do it’s sync-thang automatically without the need to call it manually.

sudo curl -L -o /bin/unison-fsmonitor
https://github.com/TentativeConvert/Syndicator/raw/master/unison-binaries/unison-fsmonitor
sudo chmod +x /bin/unison-fsmonitor

Testing it out locally

It’s possible to synchronise 2 directories on the same machine, so in the name of KISS, let’s try that out first. In this experiment we’ll try to keep the following two directories ‘in sync’:

/home/jarrod/unison/local-a
/home/jarrod/unison/local-b

Running Unison Manually

Configuration

In our Unison ‘profile’, let’s try the following so it knows ‘sup /home/jarrod/.unison/local.prf

root = /home/jarrod/unison/local-a
root = /home/jarrod/unison/local-b
auto = true
batch = true

Push the big red button

Apparently running Unison should be as simple as calling unison with the name of our profile (in this case ‘local’).

unison local

Well, that’s interesting.

Nothing to do: replicas have been changed only in identical ways since last sync.

Makes sense. Let’s create some test files and run it

jarrod@ha01:~/unison$ tree
.
|-- local-a
| |-- me
| |-- test-b
| |-- test-c
| `-- test-d
`-- local-b
 |-- test-a
 |-- test-d
 |-- test-e
 |-- test-f
 `-- test-g
2 directories, 9 files

FUCKING YES! 😄

jarrod@ha01:~/unison$ tree
.
|-- local-a
| |-- test-a
| |-- test-b
| |-- test-c
| |-- test-d
| |-- test-e
| |-- test-f
| `-- test-g
`-- local-b
 |-- test-a
 |-- test-b
 |-- test-c
 |-- test-d
 |-- test-e
 |-- test-f
 `-- test-g
2 directories, 14 files

Remove some files, and…

jarrod@ha01:~/unison$ rm local-b/test-[a-e]
jarrod@ha01:~/unison$ tree 
.
|-- local-a
| |-- test-a
| |-- test-b
| |-- test-c
| |-- test-d
| |-- test-e
| |-- test-f
| `-- test-g
`-- local-b
 |-- test-f
 `-- test-g
2 directories, 9 files
jarrod@ha01:~$ unison local
jarrod@ha01:~/unison$ tree
.
|-- local-a
| |-- test-f
| `-- test-g
`-- local-b
 |-- test-f
 `-- test-g
2 directories, 4 files

Ok. So it basically does what I want, but only when I tell it to. How can we automate this process?

Watch directories and sync automatically with Unison

Building on our above Unison profile, let’s use the following

/home/jarrod/.unison/local.prf

root = /home/jarrod/unison/local-a
root = /home/jarrod/unison/local-b
auto = true
batch = true
repeat = watch

Now, when we run Unison with our ‘local’ profile, it’ll keep running and watch the ' root’ directories for changes.

This is great!

Where it all comes crashing down 😓

The maximum number of files that can be ‘watched’ for changes by in Linux using inotify (which I can only assume unison-fsmonitor is doing) is defined by the fs.inotify.max_user_watches system variable. By default (in Debian land, at least) this number is 8192.

# cat /proc/sys/fs/inotify/max_user_watches
8192

Ok, so usually we are able to change these values - that’s the whole point of having configuration options, right?

# echo 524288 > /proc/sys/fs/inotify/max_user_watches
-su: /proc/sys/fs/inotify/max_user_watches: Permission denied

Permission fucking what?

It’s interesting to note that I’ve tried this on another “VPS” that runs under OpenVZ on my own hardware, and it appears to be a limitation of this type of virtualisation, not necessarily something that has been overridden/forbidden by a host.

Trying the same command on a KVM virtual machine is a whole ‘nother story.

# echo 8192 > /proc/sys/fs/inotify/max_user_watches
# cat /proc/sys/fs/inotify/max_user_watches
8192
# echo 524288 > /proc/sys/fs/inotify/max_user_watches
# cat /proc/sys/fs/inotify/max_user_watches
524288

Works like a bought one! 😉

To make the change to fs.inotify.max_user_watches permenant, ie not lost between reboots, you can add it to /etc/sysctl.conf.

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

TL;DR

So, this could be a viable option if A) You run your own hardware B) You aren’t running on some VM under OpenVZ

Unfortunately that’s not the situation I’m in right now 😒


Tags: #Debian #linux
Categories: Tech

‹ Previous Post Next Post ›