How to EXCLUDE copying files that start with DOT "." in Linux?

Question

I had followed several threads here in SO to copy files from one directory to another. I am using inotifywait for my purposes which is working perfectly for all BUT one scenario. It's also copying files that start with a DOT prefix (ex .tmp.swp) which I don't want.

I tried this but this even causing files with the -json postfix to NOT get copied. I don't want .tmp.abcd-json to be copied. If I remove the check AFTER the && everything is getting copied over including the .tmp.abcd-json:

These are some of the contents of the directory. The .tmp ones are unwanted but it's not ALWAYS guaranteed they will always start with .tmp. I've seen other files start with . prefixes randomly which also need to be ignored:-

abcd-json .tmp.abcd-json

#!/bin/sh
dir=/var/lib/docker/containers
target=/var/log/splunkf
inotifywait -m -r "$dir" --format '%w%f' -e create -e modify  \
| while read file;
        do
                if [[ $file == "-json"* ]] && [[ $file != "."* ]];
                then
                    echo Copying  $file to $target
                    cp -- "$file" "$target";
                else
                    echo NOT Copying  $file to $target
                fi
        done

binarysta · Accepted Answer · 2021-12-30T21:00:48.453

3

You can match with RegEx for files NOT starting with dot in your if condition as:

while read file;
        do
          f="$(basename -- $file)"
          if ! [[ "$f" =~ ^\. ]];
          then
             echo Copying  $file to $target
             cp -- "$file" "$target";
          else
             echo NOT Copying  $file to $target
          fi

edited Dec 30 '21 at 21:00

answered Jan 25 '21 at 19:28

binarysta

2,912
10
14

thanks!! The NOT was habitual (coming from C# :) ). However, this didn't work. I still see all these files which includes the undesirable! ```-rw-r-----. 1 root root 385710 Jan 25 19:40 82adf74ac58a5f29ae539b8ebbee360b21dda11e1c4767e4696ad264cb9a558f-json.log -rw-------. 1 root root 7185 Jan 25 19:40 .tmp-config.v2.json162008322``` – Dorian McAllister Jan 25 '21 at 19:42
Logs are showing it copying both the ones start with `.` and ones without. ```Copying /var/lib/docker/containers/a7079968c23e9da10a1969c7eb343a98e12ea685f8b3f3d2834666b0df5ab8c8/.tmp-hostconfig.json847502457 to /var/log/splunkf``` – Dorian McAllister Jan 25 '21 at 19:48
@DorianMcAllister yes, because the content of `$file` is the whole absolute path of file, which of course will never start with a dot and always starts with `/`. – binarysta Jan 25 '21 at 19:51
So we should search for the LAST instance of `/` and find if a file starts with `.`? any suggestions how we can do that? – Dorian McAllister Jan 25 '21 at 19:53
@DorianMcAllister yes, I updated the answer. with `basename` it's easily possible. – binarysta Jan 25 '21 at 19:59
Huzzah!! That did it. Clean and elegant@ – Dorian McAllister Jan 25 '21 at 20:11
1

Note that `! [[ $f == .* ]]` would also work. There's no need for a regular expression match. – Kusalananda Jan 25 '21 at 20:20

Kusalananda · Answer 2 · 2021-01-25T20:25:23.527

The main issue with your code is not the use of a pattern match in [[ ... ]]. It's the fact that the string that you get in $file is a pathname that contains a directory path at the start of it, i.e. the pattern .* would only ever match it if the directory path $dir started with a dot.

You also seems to run the script with /bin/sh rather than with bash, so you would not necessarily expect any of the [[ ... ]] test to work at all.

To exclude filename patterns matched by inotifywait, use --exclude 'PATTERN'. For example:

inotifywait -m -r --format '%w%f' -e create -e modify \
        --exclude '/\.[^/]*$' "$dir"

The pattern used with --exclude here matches any pathname that ends with a filename that starts with a dot. These would not be reported by inotifywait.

When using --exclude with inotifywait, your code collapses into

#!/bin/sh

dir=/var/lib/docker/containers
target=/var/log/splunkf

inotifywait -m -r --format '%w%f' -e create -e modify \
        --exclude '/\.[^/]*$' "$dir" |
xargs -I {} cp -- {} "$target"

This obviously assumes that no filename contains newlines.

Would you want to use a loop in bash with an explicit test and diagnostic output, you could use

#!/bin/bash

dir=/var/lib/docker/containers
target=/var/log/splunkf

inotifywait -m -r --format '%w%f' -e create -e modify "$dir" |
while IFS= read -r pathname; do
    if [[ ${pathname##*/} == .* ]]; then
        printf 'Not copying "%s"\n' "$pathname" >&2
    else
        printf 'Copying "%s" to "%s"\n' "$pathname" "$target" >&2
        cp -- "$pathname" "$target"
    fi
done

Note the use of IFS= read -r. This is to prevent stripping flanking whitespace from the filenames and to avoid interpreting backslash sequences (see Understanding "IFS= read -r line").

With /bin/sh, you would do

#!/bin/sh

dir=/var/lib/docker/containers
target=/var/log/splunkf

inotifywait -m -r --format '%w%f' -e create -e modify "$dir" |
while IFS= read -r pathname; do
    case ${pathname##*/} in
        .*)
            printf 'Not copying "%s"\n' "$pathname" >&2
            ;;
        *)
            printf 'Copying "%s" to "%s"\n' "$pathname" "$target" >&2
            cp -- "$pathname" "$target"
    esac
done

How to EXCLUDE copying files that start with DOT "." in Linux?

2 Answers2