How to backup/synchronize files and directories in Linux (using scp/rsync, rdiff-backup, duplicity, rsnapshot, unison, rdup, burp, dar/tar/dtrx, obnam, dd, bup, csync, dump/restore)

From backups@archlinux and file sync@wiki

  • scp@man secure copy (remote file copy program)
# install
$ sudo yum install openssh-clients | sudo apt-get install openssh-client

# remote pull, scp [OPTION...] [USER@]HOST:SRC... [DEST]
$ scp /from user@host:/to
# remote push, scp [OPTION...] SRC... [USER@]HOST:DEST
$ scp user@host:/from /to
# note: both src and dest can be remote

'-r' recursively copy entire directories
'-p' preserves modification/access times and modes
'-l limit' Limits the used bandwidth, in kbps
'-C' compression enable
'-B' batch mode (prevents asking for passwords or passphrases)
'-F ssh_config'/'-o ssh_option' use alternative ssh config file/options

from SCP examples to transfer files/folders

  • rsync@wiki/rsync@man is a file synchronization and file transfer for Unix.
    It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. Its a replacement for rcp and scp.
    It uses delta-transfer algorithm, which reduces the amount of data sent over the network. It contacts remote system using remote shell (rsh,ssh) or rsync deamon.
    Its doesnt use librsync.
## install
$ sudo yum install rsync | sudo apt-get install rsync

# local, rsync [OPTION...] SRC... [DEST]
$ rsync -az /from /to
# remote pull, rsync [OPTION...] [USER@]HOST:SRC... [DEST]
$ rsync -az /from user@host:/to
# remote push, rsync [OPTION...] SRC... [USER@]HOST:DEST
$ rsync -az user@host:/from /to
# note: both src and dest can be remote

'-a' archive mode, copying recursively, preserves symbolic links, file permissions, user & group ownerships and timestamps, equals -rlptgoD
'-r' recurse into direct­ories, but don’t preserve timestamps and permission while transferring data
'-l' copy symlinks as symlinks
'-p/-E/-A/-X/-o/-g/-t' preserve permissions/executability/ACLs/extended/owner/group/times
'-z' compress data during transfer
'--dry-run,-n' trial run, no changes

'--igno­re-­exi­sting' skip updating files that already exist on receiver
'--remo­ve-­sou­rce­-files' sender removes synchr­onized files (non-dirs)
'--delete' delete extraneous files from destin­ation dirs
'--partial' keeps partially transfered files
'--existing' skip creating new files on receiver
$ rsync -az --delete user@host:/from /to
$ rsync -az --remove-source-files backup.tar /to

'--bwlimit=KBPS' limit I/O bandwidth; KBytes per second
'--exclude=PATTERN' exclude files matching PATTERN
'--incl­ude­=PA­TTERN' don't exclude files matching PATTERN
$ rsync -az --include 'R*' --exclude '*' user@host:/from /to

'-b' preexisting destination files are renamed as each file is transferred or deleted
'-u' skip files newer in destination

## run rsync daemon, using systemd; already included in latest package
$ cat /{etc,usr/lib}/systemd/system/rsyncd.socket
; socket activation to start on first connect, systemctl enable rsyncd.socket
[Unit]
Description=Rsync Server Socket
Conflicts=rsyncd.service
[Socket]
ListenStream=873
Accept=yes
[Install]
WantedBy=sockets.target 
$ cat /{etc,usr/lib}/systemd/system/rsyncd.service
; daemon to start at boot, systemctl enable rsyncd.service
[Unit]
Description=fast remote file copy program daemon
ConditionPathExists=/etc/rsyncd.conf
[Service]
EnvironmentFile=/etc/sysconfig/rsyncd
ExecStart=/usr/bin/rsync --daemon --no-detach "$OPTIONS"
[Install]
WantedBy=multi-user.target
$ rsync localhost::

from rsync cheat sheet

  • rdiff-backup/rdiff-backup@man local/remote mirror and incremental backup.
    Uses librsync (also used by rdiff@man, a diff/patch but in a different delta format) instead of rsync.
    Stores the most recent backup as regular files, To revert to older versions, you apply the diff files to recreate the older versions.
# install
$ sudo yum install rdiff-backup | sudo apt-get install rdiff-backup

# backup 'local-dir' to remote dir which will also include 'rdiff-backup-data' to restore previous states
$ rdiff-backup local-dir user@host.net::/remote-dir

# restore current version, '-r/--restore-as-of'
$ rdiff-backup -r now host.net::/remote-dir/file local-dir/file
# restore from 10 days ago
$ rdiff-backup -r 10D host.net::/remote-dir/file /tmp/file

# delete older files, '--remove-older-than'
$ rdiff-backup --remove-older-than 2W host.net::/remote-dir

# file selection include/exclude; '**' matches any path, '*' matches any path except '/'
$ rdiff-backup --exclude /tmp --exclude /mnt --exclude /proc user@host.net::/ /backup/host.net

# list number and date of partial incremental backups from given directory/file, '-l/--list-increments'
$ rdiff-backup -l out-dir/file
# list all files in directory changed since given date, '-list-changed-since'
$ rdiff-backup --list-changed-since 5D out-dir/subdir
# list all files present at given time, '--list-at-time'
$ rdiff-backup --list-at-time 5D out-dir/subdir

# compare changes in directories
$ rdiff-backup --compare in-dir user@host::out-dir
# same by as seen by date
$ rdiff-backup --compare-at-time 2W in-dir user@host::out-dir

from rdiff-backup examples

  • duplicity/duplicity@man backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server.
    Its a variation on rdiff-backup that allows for backups without cooperation from the storage server.
    Uses librsync to build space efficient incremental archives.
    Supports deleted files, full Unix permissions, directories, symbolic links, fifos, etc., but not hard links.
    Supports local, scp/ssh, sftp, rsync, ftp, s3, webdav: ‘scheme://user[:password]@host[:port]/[/]path’.
# install
$ sudo apt-get install duplicity | sudo yum install duplicity (EPEL)

# backup '/from' using scp/ssh to remote; repeating command will make incremental backups
$ duplicity /from scp://user@host//usr/backup
# force full backup 'full/incremental'
$ duplicity full --exclude /tmp /from scp://user@host//usr/backup

# restore
$ duplicity scp://user@host//usr/backup restored_dir

from How to create a secure incremental offsite backup in Linux with Duplicity

  • rsnapshot/rsnapshot@man remote filesystem snapshot utility.
    It can take incremental snapshots of local and remote filesystems for any number of machines.
## install
$ sudo yum install rsnapshot | sudo apt-get install rsnapshot | sudo pacman -S rsnapshot

## configure
$ cat /etc/rsnapshot.conf
...
# local filesystem path to save all snapshots
snapshot_root /mnt/backups/
...
# number of snapshots to retain by 'type'
retain hourly 6
...
# backup /etc/ to <snapshot_root>/<retain>.0/localhost/etc/ using rsync on the local filesystem
backup /usr/local/ localhost/
# backup /usr/local/ to <snapshot_root>/<retain>.0/localhost/usr/local/ using rsync on the local filesystem
backup /usr/local/ localhost/
#backup root@example.com:/etc/ to <snapshot_root>/<retain>.0/example.com/etc/ using rsync over ssh
backup root@example.com:/etc/ example.com/

# test
$ rsnapshot configtest
# verify by type
$ rsnapshot -t hourly

# schedule backups using cron
$ crontab -e
0 */4 * * * /usr/local/bin/rsnapshot hourly
30 23 * * * /usr/local/bin/rsnapshot daily
(or using systemd)
$ cat /{etc,usr/lib}/systemd/system/rsnapshot@.service
[Unit]
Description=rsnapshot (%I) backup
[Service]
Type=oneshot
Nice=19
IOSchedulingClass=3
ExecStart=/usr/bin/rsnapshot %I
$ cat /{etc,usr/lib}/systemd/system/rsnapshot-daily.timer
[Unit]
Description=rsnapshot daily backup
[Timer]
# 14:30 is the clock time when to start it
OnCalendar=14:30
Persistent=true
Unit=rsnapshot@daily.service
[Install]
WantedBy=timers.target
$ systemctl enable rsnapshot-daily.timer ; systemctl start rsnapshot-daily.timer

from rsnapshot@archlinux and rsnapshot@tecmint

  • unison/unison@wiki supports bi-directional file synchronization using a smart diff method + rsync.
    Allows the user to interactively choose which changes to push, pull, or merge.
## install
$ sudo yum install unison | sudo apt-get install unison | sudo pacman -S unison

# sync '/path' with a remote interactively
$ unison /path ssh://host//path

## using profiles, non-interactively
$ cat ~/.unison/default.prf
# roots of the synchronization, supports ssh://,rsh://, socket://
root = /sync_folder
root = ssh://dev@192.168.1.10//sync_folder (under the root)
# path = dir1
# regexps specifying names and paths to ignore
#ignore = Path stats/*  ## ignores /stats/*
#ignore = Name *stats   ## ignores all files/directories that end with "stats"
# no question asked, non-conflicting changes will be propagated, conflicts will be skipped.
auto = true
# source for force one-way sync
# force = /sync_folder
$ unison (in either server)

from How to synchronize files between two servers bidirectionally

  • rdup scripts to facilitate backups and delegates the encryption, compression, transfer and packaging to other utilities.
# install
$ sudo yum install rdup (RPMforge) | sudo apt-get install rdup

# prints list of files that changed, or all in case of /dev/null dump
$ rdup -N timestamp LIST DIR

# rdup mirroring, backup
$ rdup /dev/null ~/bin | rdup-up -t /shared/backup
# and restore
$ rdup /dev/null /shared/backup | rdup-up -t /tmp/restore

# rdup archiving, backup
$ rdup /dev/null ~/bin > my-archive.rdup
# and restore
$ rdup-up -t /tmp/restore  /mnt/disk1/sda.img.gz
# restore
$ gunzip -c /mnt/disk1/sda.img.gz | sudo dd of=/dev/sda

# backup/restore to/from a remote
$ sudo dd if=/dev/sda | gzip -c | ssh user@remote_host "cat > /mnt/disk1/sda.img.gz"
$ ssh user@remote_host "cat /mnt/disk1/sda.img.gz" | gunzip -c | sudo dd of=/dev/sda

from How to backup a hard disk on Linux

  • bup@github/bup@ubuntu based on the git packfile format, providing fast incremental saves and global deduplication.
    Uses a rolling checksum algorithm (similar to rsync) to split large files into chunks.
# install
$ sudo apt-get install bup
# for rhel/centos build from source, https://build.opensuse.org/package/show/home:p_conrad:branches/bup
$ yum install -y python-devel git-core python-fuse && rpmbuild -bb burp.spec

# local backup; '-x' to limit to one filesystem, '--exclude' to exclude paths, '-u' update, index, default option
$ bup init ; bup index -x /etc ; bup save -n local-etc /etc

# restore local backup to ./dest; '-n' branch name
$ bup restore -C ./dest local-etc/latest/etc
# make another backup, incremental
$ bup index /etc ; bup save -n local-etc /etc
# restore previous backup
$ bup restore -C ./dest-2 local-etc/2013-11-23-11195/etc

# list previous backups
$ bup ls local-etc

# backup to remote; '-r/--remote=host:port' using ssh
$ ssh host bup init; bup index /etc ; bup save -r host: -n local-etc /etc
# restore from remote; 'save -r' isnt supported, so
$ sudo sshfs user@host:/dest-2 ./dest-2 ; ssh host bup restore -C ./dest-2

# use split/join instead of index/save/restore; create a local backup using tar
$ tar -cvf - /etc | bup split -n local-etc -vv
# and restoring
$ bup join local-etc | tar -tf -
# create another "incremental" backup
$ tar -cvf - /etc | bup split -n local-etc -vv
# restoring the previous backup
$ bup join local-etc~1 | tar -tf -

# backup on a remote
$ tar -cvf - /etc | bup split -r SERVERNAME: -n local-etc -vv
# restoring from a remote
$ bup join -r SERVERNAME: local-etc | tar -tf -
  • csync lightweight bidirectionally utility to synchronize files between two directories on a system or between multiple systems. Supports sftp and smb.
    Used by (OwnCloud)[http://en.wikipedia.org/wiki/OwnCloud].
# install, for fedora see http://download.opensuse.org/repositories/network:/synchronization:/files/
$ sudo add-apt-repository ppa:markhannon/ppa && sudo apt-get update && sudo apt-get install csync

csync [OPTION...] SOURCE DESTINATION
$ csync replica1 relplica2
# remote 'sftp://' or 'smb://'
$ csync replica1 smb://user:passwd@host/replica2 or csync replica1 sftp://user:passwd@host/replica2

from Csync – Client only bidirectional file synchronizer and CSYNC User Guide

  • dump@man/restore@man ext2/3 filesystem backup.
    Doesn't work on subdirectories, only filesystems (and only on ext2/3).
# install
$ sudo apt-get install dump | sudo yum install dump

# exclude pahs
$ chattr +d /path
$ lsattr -d /path
------d------e- /path

'-level#' dump level, '0' is full backup, otherwise incremental backup (copy all files new or modified since the last dump of a lower level) 
'-h level' honor the user nodump flag, default honor level is 1
'-u' update the file /etc/dumpdates after a successful dump
'-f file' write the backup to file; file may /dev/st0 (a tape drive), /dev/sda1 (a disk drive), an ordinary file, or - (the standard input)

# full backup, into '/dev/sa0' tape
$ dump -0u -f /dev/st0 /path
# incremental backup
$ dump -1u -f /dev/st0 /path

'-t' list and '-C' compare to current
'-r' restore/rebuild filesystem
'-x' extract individual files
'-f file' read the backup from file; file may /dev/st0 (a tape drive), /dev/sda1 (a disk drive), an ordinary file, or - (the standard input)
$ restore -Cf /dev/st0
$ restore -rf /dev/st0

from UNIX / Linux Dump Command Exclude Directories / Files and Linux Tape Backup With mt And tar Command Howto

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s