Ordering files transferred by rsync

Question

We're currently using rsync to run a backup job. This job takes several hours. As there are some parts of the backup that are more important than others, we would like to ensure that they are backed up before everything else (i.e. that rsync transfers those files first). This should be easily accomplished using rsync --files-from.

Unfortunately, the files that we need to prioritise are alongside all of the other files in a single directory (flat hierarchy, boo!), they just have a specific extension to identify them. As such, we'd like to get a list of the files that rsync would transfer, sort it locally and then pass it to rsync using rsync --files-from.

Using rsync -v --dry-run gets us most of the way there (I think), but displays directory-additions and file property changes, meaning we don't really get an immediately usable list.

So, my question is this: how can I get a list of files to use from rsync?

Answer 1

Would running rsync twice work? i.e. run it once for the high-priority files and once for the rest?

Answer 2

I would have commented on @alberge's answer but I don't have enough reputation :-(

@Daniel, if you are using separate rsync commands might I make a suggestion to use a for loop:

for pp in "*.png" "*.jpg" "*.bmp" "*.gif"
do
    rsync ... $pp
done

just one rsync command to alter should you want to make any changes and a nice easy managable list of things to sync. The order in the for statement is the order in which they are used.

Answer 3

You could use rdiff-backup first, or just replace rsync with rdiff-backup. For example, you can get you list of files with:

rdiff-backup --compare in-dir user@host::out-dir

However, I'd go with the other suggestion to try two passes. Also, have you tried enabling compression?