rsync pass multiple exclude from $@

Question

I'm trying to make a wrapper around my rsync command to reuse code. I need to pass the source folder, the target folder and then any number of excluded directories. Here is what I have with the logs; basically I am building my multiple rsync's --exclude params from my "vararg" param. There may be a better way but I found that one ok enough.

function run_backup(){
    source="$1"
    target="/backups/unraid$2"
    #Copying $source into remote folder $target

    # Building my exclude params
    if [ $# -ge 3 ]; then
        excludes="--exclude '$3'"
        while shift && [ -n "$3" ]; do
            echo $excludes
            excludes="${excludes} --exclude '$3'"
        done
    else
        excludes = "--exclude '/tmp'" # Dummy exclude; I'll handle this case later
    fi
    echo $excludes
    
    # Creating folder because rsync crash otherwise n some case
    ssh -p 22 -i /home/backup/.ssh/mykey [email protected] "mkdir -p $target"

    # Executing the copy
    set -e
    sudo rsync -azvh \
        -e "ssh  -p 22-i /home/backup/.ssh/mykey "\
        --progress \
        --delete \
        $excludes \
        $source \
        [email protected]:$target
}


mkdir -p /tmp/dummy
echo "dummy" > /tmp/dummy/save-ignored-1
echo "dummy" > /tmp/dummy/save-ignored-2
echo "dummy" > /tmp/dummy/save-test
run_backup /tmp/dummy/ /dummy /tmp/dummy/save-ignored-1 /tmp/dummy/save-ignored-2 /tmp/toto /tmp/tata /tmp/tutu #TODO

--exclude '/tmp/dummy/save-ignored-1'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2' --exclude '/tmp/toto'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2' --exclude '/tmp/toto' --exclude '/tmp/tata'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2' --exclude '/tmp/toto' --exclude '/tmp/tata' --exclude '/tmp/tutu'
++ ssh -p 22-i /home/backup/.ssh/mykey [email protected] 'mkdir -p /backups/unraid/dummy'
++ sudo rsync -azvh -e 'ssh  -p 22 -i /home/backup/.ssh/mykey' --progress --delete --exclude ''\''/tmp/dummy/save-ignored-1'\''' --exclude ''\''/tmp/dummy/save-ignored-2'\''' --exclude ''\''/tmp/toto'\''' --exclude ''\''/tmp/tata'\''' --exclude ''\''/tmp/tutu'\''' /tmp/dummy/ [email protected]:/backups/unraid/dummy

We can see the $excludes var contain exactly what I want; but bash is doing some magic on it when I pass it to rsync. And I have no idea how to avoid that. I guess it has to do with expansion, but I could not find a problem similar to mine or the revelant doc.

EDIT: code after first answer:

function run_backup(){
    excludes=()
    if [ $# -ge 3 ]; then
        echo $3
        excludes+=(--exclude $3)
        while shift && [ -n "$3" ]; do
            echo $3
            excludes+=(--exclude $3)
        done
    else
        excludes+=(--exclude /tmp)
    fi
    set -x
    echo "${excludes[@]}"
}
run_backup /tmp/dummy/ /dummy "/tmp/dummy/save-ignored" "/tmp/dummy/save non'standard ignored" /tmp/toto /tmp/tata /tmp/tutu

But it won't work for string containing quotes (at least simple quotes)

/tmp/dummy/save-ignored
/tmp/dummy/save non'standard ignored
/tmp/toto
/tmp/tata
/tmp/tutu
++ echo --exclude /tmp/dummy/save-ignored --exclude /tmp/dummy/save 'non'\''standard' ignored --exclude /tmp/toto --exclude /tmp/tata --exclude /tmp/tutu
--exclude /tmp/dummy/save-ignored --exclude /tmp/dummy/save non'standard ignored --exclude /tmp/toto --exclude /tmp/tata --exclude /tmp/tutu

EDIT 2: Useless; just keeping it for history ~~A test with all quotations possibilities:~~

run_backup(){
    excludes=()
    excludes2=()
    excludes3=()
    if [ $# -ge 3 ]; then
        echo $3
        excludes+=(--exclude $3)
        excludes2+=(--exclude "$3")
        excludes3+=(--exclude "\"$3\"")
        while shift && [ -n "$3" ]; do
            echo $3
            excludes+=(--exclude $3)
            excludes2+=(--exclude "$3")
            excludes3+=(--exclude "\"$3\"")
        done
    else
        excludes+=(--exclude /tmp)
    fi
    set -x
    echo "${excludes[@]}"
    echo "${excludes2[@]}"
    echo "${excludes3[@]}"
     sudo rsync -azvh \
        --dry-run \
        "${excludes[@]}" \
        --dry-run \
        "${excludes2[@]}" \
        --dry-run \
        "${excludes3[@]}" \
        --dry-run \
        ${excludes[@]} \
        --dry-run \
        ${excludes2[@]} \
        --dry-run \
        ${excludes3[@]} \
        /tmp \
        [email protected]:/tmp
}
run_backup "/tmp/dummy/" "/dummy" "/tmp/dummy/save non'standard" "/tmp/toto"

++ echo --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto
--exclude /tmp/dummy/save non'standard --exclude /tmp/toto
++ echo --exclude '/tmp/dummy/save non'\''standard' --exclude /tmp/toto
--exclude /tmp/dummy/save non'standard --exclude /tmp/toto
++ echo --exclude '"/tmp/dummy/save non'\''standard"' --exclude '"/tmp/toto"'
--exclude "/tmp/dummy/save non'standard" --exclude "/tmp/toto"
++ sudo rsync -azvh --dry-run --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto --dry-run --exclude '/tmp/dummy/save non'\''standard' --exclude /tmp/toto --dry-run --exclude '"/tmp/dummy/save non'\''standard"' --exclude '"/tmp/toto"' --dry-run --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto --dry-run --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto --dry-run --exclude '"/tmp/dummy/save' 'non'\''standard"' --exclude '"/tmp/toto"' /tmp [email protected]:/tmp

Screenshot of my test with better formated outputs:

See also: http://askubuntu.com/questions/320458/how-to-exclude-multiple-directories-with-rsync & http://askubuntu.com/questions/545655/backup-your-home-directory-with-rsync-and-skip-useless-folders & http://askubuntu.com/questions/40992/what-files-and-directories-can-be-excluded-from-a-backup-of-the-home-directory/40997#40997 — oldfred, Aug 31 '22 at 17:52
this, `excludes="--exclude '$3'"`, puts literal quotes in the value of `excludes`, they'll stay literal as the shell doesn't parse the results of expansions for shell syntax (it would be a horrible safety issue, e.g. it'd never be possible to deal with strings like `ain't so` in a variable). Even if it did work, it'd fail for values that themselves contain single quotes. The subsequent `$excludes` will then wordsplit on whitespace, breaking any excluded paths that contain whitespace. Use an array instead since you're running Bash, which supports them. — ilkkachu, Aug 31 '22 at 20:17
Also (you probably know this), there shouldn't be any spaces around the `=` in `excludes = "--exclude '/tmp'"`. Also the `function` keyword is unnecessary and non-standard, just `run_backup() { ...` would do to define a function — ilkkachu, Aug 31 '22 at 20:18
See [But what if …?](https://unix.stackexchange.com/q/171346/80216#286350) — G-Man Says 'Reinstate Monica', Sep 04 '22 at 00:53
Please learn how to debug. You should have been able to track your problem down to the ``excludes="--exclude '$3'"`` line; then you could have spared us the 30-line code block, and you would have been that much closer to asking a straightforward question — and, ideally, finding a solution yourself. And your “EDIT 2” is terrible. You use four arguments, totaling 64 characters, and three of them are equivalent (strings of letters and slashes), and the fourth one has two special features (space and quote). Simplify! You present a 500+ character output and make no attempt to analyze it. Etc… — G-Man Says 'Reinstate Monica', Sep 04 '22 at 00:53
You should be using `excludes+=(--exclude "$3")`. Quote your shell parameter expansions, please. — muru, Sep 04 '22 at 07:23
@G-ManSays'ReinstateMonica' I agree with some of this. My first question I did simplify from the orgiginal code, maybe I could have done more but I had no idea where the issue was comming from and didn't want to remove too much. My first edit is ok IMO. But my second edit is shit; I wrote it while being completly lost and wanted to show how no answer given where ok ina single example - until I find the answers where working but I did not understood how it actually worked. I explained that in my las comment to laktak and have now edited my question to avoid people the trouble of reading it. — Bancarel Valentin, Sep 05 '22 at 09:09

score 4 · Accepted Answer · answered Aug 31 '22 at 17:35

4

Use an array instead of a string.

For example:

opt=()
opt+=(--exclude "first")
opt+=(--exclude "another one")

set -x
echo "${opt[@]}"

This will handle spaces correctly (output):

+ echo --exclude first --exclude 'another one'

answered Aug 31 '22 at 17:35

laktak

5,616
20
38

This is perfect for the use case described. But I actuallt would like to be able to handle files with spaces or quotes in their names. I have tried adding quotes (simple and/or double) at different places but couldn't find a fix. I have edited my answer with snippet you gave me and the log it produces to show you. Any idea of hoow to resolve that ? – Bancarel Valentin Aug 31 '22 at 21:54
@BancarelValentin it should already be handling such names. If it's not please add to your question the _exact_ code you're now using with an explanation of what's not working – roaima Aug 31 '22 at 22:30
Yup, I did exactly taht already if you want to have a look. – Bancarel Valentin Aug 31 '22 at 22:44
@BancarelValentin in your updated code you need to use quotes like in my answer, e.g. `excludes+=(--exclude "$3")` – laktak Sep 01 '22 at 19:38
@laktak with that, one of my excluse looks like this `--exclude /tmp/dummy/save non'standard ignored`. There is a quote messing up here – Bancarel Valentin Sep 02 '22 at 09:41
I should say I also tried `excludes+=(--exclude "\"$3\"")`; with that the echo looks great `--exclude "/tmp/dummy/save non'standard ignored"` but not the actual rsync command: `sudo rsync -azvh --delete --stats --exclude '"/tmp/dummy/save non'\''standard ignored"'` – Bancarel Valentin Sep 02 '22 at 15:26
When I use your sample with `"$3"` I get this: `+ echo --exclude /tmp/dummy/save-ignored --exclude '/tmp/dummy/save non'\''standard ignored' --exclude /tmp/toto --exclude /tmp/tata --exclude /tmp/tutu` which is correct – laktak Sep 02 '22 at 19:32
1

You are correct; I just edited my answer and was about to post more details but I just missed your last point `'/tmp/dummy/save non'\''standard ignored'` is indeed correct. I did not understood there was a concatenation in the middle there. – Bancarel Valentin Sep 03 '22 at 17:00

rsync pass multiple exclude from $@

1 Answers1