This blog has moved to Medium

Subscribe via email


Easy linux backup with duplicity

After looking at several backup solutions for a linux-based project of mine, ranging from more advanced systems like bacula to a range of python/bash scripts, I finally found the ultra simple duplicity. Don’t get me wrong, bacula and its brethren really seem more powerful overall, but when you just need a simple solution, you can’t compete with duplicity’s simplicity.

duplicity is drop-dead simple backup system that does incremental/full backups, and supports a wide variety of backup targets from the box, including file, ftp, scp, s3, and a dozen others. The usage is simply

duplicity /folder/to/backup file:///path/to/target/folder

You can have the target folder be a local or S3-mounted folder (with the excellent s3fs), or use other supported URIs. The default is an incremental backup, by you can override by choosing a full backup. It even has support for “keep the last N full backups and delete the rest”. Update – after running some time with backups into s3fs, I recommend against it – it gave us some performance headaches. Use the native s3 option that duplicity has instead.

The only feature that I required, and duplicity didn’t have, is automatically choosing when to do full vs automatic backups. I would like to tell it “every 100 runs, do a full backup”, and this is not supported out of the box … although easily fixed with a wrapper script. Put that in crontab and you’re set. Oh, by default it requires setting a PGP key, but if you’re lazy you can skip it with the –no-enc option.

8 Comments

  1. Ronen:

    I think that the default backup utility in Ubuntu uses that also, point it to an ssh server and thats it,

    The Ubuntu UI client does do a full backup once in a while

  2. ripper234:

    @Ronen – what’s the default backup utility? I’m talking about an Ubuntu server on EC2, it doesn’t have the UI sub-system as far as I know.

  3. Ronen:

    Yeap Im talking about the desktop version (its installed by default):

    https://launchpad.net/deja-dup

    It uses duplicity behind the scenes

  4. Ronen:

    BTW you can tell it to do full after some time:

    –full-if-older-than time
    Perform a full backup if an incremental backup is requested, but the latest full backup in the collection is older than the given time. See the TIME FORMATS section for more information.

  5. ripper234:

    @Ronen – very nice, I missed that flag, it simplifies my life considerably 🙂

  6. Sultan:

    How does it compare to http://www.boxbackup.org/ (if you happen to have done the comparison during your research). I’m currently looking for a system, too.
    Thanks

  7. ripper234:

    @Sultan – I don’t know, I don’t remember finding boxbackup in my research.
    If you try it, please post about your experience.

    Duplicity has been doing a great job for us – just be sure not to use s3fs as I wrote above, but rather use the builtin S3 connector – we ran into some perf problems with duplicity on s3fs.

  8. Sultan:

    @ripper234: Thanks. I’ll look more into Boxbackup (and Duplicity) and try to remember to post back when I do. It won’t be for a while, though. Very busy these days… .