I need to setup a synchronization between my Computers with a NAS in the local network. The setup is somewhat complicated, though, so I am assuming there is not an off-the-shelf product that does what I need.

Details:

Devices I use:

• Desktop PC, dual-boot Windows 10 / Arch Linux
• Laptop (Arch Linux)
• NAS (self-build, FreeNAS running on it)

I've used rsync so far to backup to the NAS (one-way, my dual-boot setup uses the same storage for both OS).

The problem is, I would like to get the Laptop in the sync. Therefore I need to do some kind of two-way sync.

Problems that arise here:

1. I often save files only seconds before the shutdown, so I need to monitor file changes instead of running a sync tool periodically

   • I cannot sync my laptop outside of the local network (and in the local network, I use the desktop PC most of the times); the local (updated) version could be quite out of date and give some conflicts.

2. I do want to use the local network only (probably the existing network share of the NAS), not an ownCloud or something similar, because we have quite limited bandwidth where I live and I have some rather huge files to keep synced.

My ideas

I found this question: How do I setup rsync and any other scripts to mimic a minimal viable replica of DropBox?, and looked into a solution using lsyncd (Live Syncing Daemon), which synchronizes local directories with remote targets. This could solve problem #1, but I still need to be prepared for conflicts. Furthermore, I could not find any docs about lsyncd on Windows, so I assume it won't work there.

I also looked at the potential to incorporate Unison into a solution; it seems to do two-way syncs. Maybe I could use it together with lsyncd. I'm not sure about the conflict handling, though. I found a hint that with -batch, conflicts are just skipped. But how do I handle conflicted files in the long term (I want to sync them, after all)?

So my question is, how can I accomplish this?

3

2 Answers

Depending on your NAS - I'm pretty fond of BitTorrent sync for this. It runs on most systems (including raspberry pis and mips systems) with support from quite a few consumer NAS systems (like my seagate). You just pick a folder to share, generate a key, and use the key to set up the share on each system you want it on. Its smart enough to detect you're on a LAN, works outside of it. The free version works well enough for my needs - I mainly use it to sync downloads from a system I keep on all the time and backups of many small files that I add to infrequently.

It'll do what you need - one way or two way sync, using the lan if needed. It deals with conflict by keeping a copy of deleted files, (which I turn off for my use cases) and generally runs well enough that I hardly notice it.

2

So, my self-developed solution requires a bit more scripting than the one of Journeyman Geek. I used unison for the sync-part and zenity to warn for conflicts.

Scripts

The bash script (put this into your startup-applications):

#!/bin/bash
if [ -z "$(arp | grep MAC_ADDRESS_OF_NAS)" ]; then zenity --warning --text="NAS not found. Aborting."
fi;
unison sync -batch=true -repeat=watch |& while read -r line; do if test -n "$(echo $line | grep skipped:)"; then zenity --warning --text="$line"; fi;
done

The unison configuration (put this as sync.prf into ~/.unison/):

root=/home/my-username/the-data-i-want-to-backup/
root=/mnt/NAS/some/sub/path/
# follow all symlinks
follow = Path *
# automatically do everything (if there are no conflicts)
auto = true
# for some reason unison fails if I sync the permissions
perms = 0
dontchmod = true

Detailed explanation (and troubleshooting for derivates)

1. First I check, if the NAS is found in the local network. If its MAC-address does not appear in the arp output, I assume I don't need to sync. This needs probably some further tweaking as I rarely turn my laptop off completely, so the script won't be started when I connect to the network. Maybe I'll add a cron job or some kind of "on-network-connect"
2. zenity can be used to create graphical dialogs from a bash script. You probably need to install it. Basic usage is pretty straight forward.
3. unison sync starts the synchronization. It looks for a configuration profile called ~/.unison/sync.prf. I could substitute sync by something else (both in the filename and the command).
4. To skip conflicted files, I specify -batch=true. I don't do this in the config file because I want to run unison sync manually if a conflict happens.
5. I don't need a third party tool to watch for changes: -repeat=watch does exactly what I want.Sidenote: actually it starts a full resync on any change, so it could do better, but that's ok for me.
6. The output of the unison-command is a bit strange, took some time to get this working:
   • If there's a conflict, the output contains skipped: FILENAME
   • That seems to happen on STDERR rather than on STDOUT, so I need to pipe with |&
   • For some reason I could not figure out it did not work with a simple grep for me. I tried something like unison sync ... |& grep "skipped:" | xargs zenity --text="{}", but that would not execute until unison finally finished (it worked fine without -repeat=, and it worked fine without the grep)
7. For above reasons I sticked with a while loop to read the input and use the grep only on each line itself
8. In the unison profile, notice that I used absolute paths. For some reason it did not work with ~/my/folder

Additional notes

• I'm someone who tends to over-engineer. I setup a script generating me the unison-profile. In case you're interested:

  #!/bin/bash
  HOME_DIR=$(cd ~; pwd)
  REPO_DIR="$(dirname "$(readlink -f $0)")"
  UNISON_PRF=$HOME_DIR/.unison/sync.prf
  # write sync script
  read -e -p "Local sync dir: [like ~/Data] " LOCAL_SYNC_DIR
  # make path absolute:
  LOCAL_SYNC_DIR="${LOCAL_SYNC_DIR/#\~/$HOME_DIR}"
  read -e -p "Remote sync dir: [like /mnt/NAS/Data/] " REMOTE_SYNC_DIR
  REMOTE_SYNC_DIR="${REMOTE_SYNC_DIR/#\~/$HOME_DIR}"
  rm "$UNISON_PRF"
  echo "root=$LOCAL_SYNC_DIR" > "$UNISON_PRF"
  echo "root=$REMOTE_SYNC_DIR" >> "$UNISON_PRF"
  cat "$REPO_DIR/sync.prf" >> "$UNISON_PRF"
  # user info
  echo "IMPORTANT: You need to put $REPO_DIR/unison/sync.sh to your startup applications for this to work properly!"

0

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy