File Syncronisation with Unison

What was supposed to be a quick-and-dirty on 2-way file synchronisation for Debian servers using Unison.

Homepage: https://www.cis.upenn.edu/~bcpierce/unison/
GitHub: https://github.com/bcpierce00/unison
Manual: https://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html

See Also: https://medium.com/@yobi.kim/how-to-install-unison-91c9fb36ac09

Installation

From Debian Repos

Assuming a Debian server, we can install Unison using apt-get.

1

sudo apt-get update && sudo apt-get install unison

Currently (2018/09/03) this will install the following version of Unison, depending on the version of Debian you’re running:

• 8 Jessie: v2.40.102
• 9 Stretch: v2.48.3

From source

If a later release than that published in the Debian repositories is required, we can build it from source.
Grab the latest release source code from https://github.com/bcpierce00/unison/releases

Make sure we have the required OCaml compiler stuff

1

sudo apt-get install ocaml-native-compilers

Then build it (replacing 2.51.2 with whatever is the latest release)

1
2
3
4

wget https://github.com/bcpierce00/unison/archive/v2.51.2.tar.gz
tar -zxf v2.51.2.tar.gz
cd unison-2.51.2/
make all && make test && sudo HOME=/usr make install

Hmm…

Looks like neither of the above installation methods provide us with the unison-fsmonitor tool, which would allow us to run Unison kinda as a daemon/service with some tweaks.
The unison-fsmonitor tool can watch a directory for file changes, and trigger Unison to do it’s sync-thang automatically without the need to call it manually.

1
2
3

sudo curl -L -o /bin/unison-fsmonitor
https://github.com/TentativeConvert/Syndicator/raw/master/unison-binaries/unison-fsmonitor
sudo chmod +x /bin/unison-fsmonitor

Testing it out locally

It’s possible to synchronise 2 directories on the same machine, so in the name of KISS, let’s try that out first.
In this experiment we’ll try to keep the following two directories ‘in sync’:

1
2

/home/jarrod/unison/local-a
/home/jarrod/unison/local-b

Running Unison Manually

Configuration

In our Unison ‘profile’, let’s try the following so it knows ‘sup
/home/jarrod/.unison/local.prf

1
2
3
4

root = /home/jarrod/unison/local-a
root = /home/jarrod/unison/local-b
auto = true
batch = true

Push the big red button

Apparently running Unison should be as simple as calling unison with the name of our profile (in this case ‘local’).

1

unison local

Well, that’s interesting.

1

Nothing to do: replicas have been changed only in identical ways since last sync.

Makes sense. Let’s create some test files and run it

1
2
3
4
5
6
7
8
9
10
11
12
13
14

jarrod@ha01:~/unison$ tree
.
|-- local-a
| |-- me
| |-- test-b
| |-- test-c
| `-- test-d
`-- local-b
 |-- test-a
 |-- test-d
 |-- test-e
 |-- test-f
 `-- test-g
2 directories, 9 files

FUCKING YES! 😄

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

jarrod@ha01:~/unison$ tree
.
|-- local-a
| |-- test-a
| |-- test-b
| |-- test-c
| |-- test-d
| |-- test-e
| |-- test-f
| `-- test-g
`-- local-b
 |-- test-a
 |-- test-b
 |-- test-c
 |-- test-d
 |-- test-e
 |-- test-f
 `-- test-g
2 directories, 14 files

Remove some files, and…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

jarrod@ha01:~/unison$ rm local-b/test-[a-e]
jarrod@ha01:~/unison$ tree 
.
|-- local-a
| |-- test-a
| |-- test-b
| |-- test-c
| |-- test-d
| |-- test-e
| |-- test-f
| `-- test-g
`-- local-b
 |-- test-f
 `-- test-g
2 directories, 9 files
jarrod@ha01:~$ unison local
jarrod@ha01:~/unison$ tree
.
|-- local-a
| |-- test-f
| `-- test-g
`-- local-b
 |-- test-f
 `-- test-g
2 directories, 4 files

Ok. So it basically does what I want, but only when I tell it to.
How can we automate this process?

Watch directories and sync automatically with Unison

Building on our above Unison profile, let’s use the following

/home/jarrod/.unison/local.prf

1
2
3
4
5

root = /home/jarrod/unison/local-a
root = /home/jarrod/unison/local-b
auto = true
batch = true
repeat = watch

Now, when we run Unison with our ‘local’ profile, it’ll keep running and watch the ‘ root‘ directories for changes.

This is great!

Where it all comes crashing down 😓

The maximum number of files that can be ‘watched’ for changes by in Linux using inotify (which I can only assume unison-fsmonitor is doing) is defined by the fs.inotify.max_user_watches system variable. By default (in Debian land, at least) this number is 8192.

1
2

# cat /proc/sys/fs/inotify/max_user_watches
8192

Ok, so usually we are able to change these values - that’s the whole point of having configuration options, right?

1
2

# echo 524288 > /proc/sys/fs/inotify/max_user_watches
-su: /proc/sys/fs/inotify/max_user_watches: Permission denied

Permission fucking what?

It’s interesting to note that I’ve tried this on another “VPS” that runs under OpenVZ on my own hardware,
and it appears to be a limitation of this type of virtualisation, not necessarily something that has been overridden/forbidden by a host.

Trying the same command on a KVM virtual machine is a whole ‘nother story.

1
2
3
4
5
6

# echo 8192 > /proc/sys/fs/inotify/max_user_watches
# cat /proc/sys/fs/inotify/max_user_watches
8192
# echo 524288 > /proc/sys/fs/inotify/max_user_watches
# cat /proc/sys/fs/inotify/max_user_watches
524288

Works like a bought one! 😉

To make the change to fs.inotify.max_user_watches permenant, ie not lost between reboots, you can add it to /etc/sysctl.conf.

1

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

TL;DR

So, this could be a viable option if
A) You run your own hardware
B) You aren’t running on some VM under OpenVZ

Unfortunately that’s not the situation I’m in right now 😒