Check if file exists in S3 Bucket

Question

This directory /data/files/ has thousands files like:

1test
2test
3test

[...]

60000test
60001test

I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be offline and because of that the file is skipped.

How can I check if the file that exists in /data/files/ is also in the S3 Bucket? and if not copy the missing file to S3?

I would prefer to do this using BASH. Also if I need to change the AWS CLI for another one, can be.

There are a bunch of command-line tools that talk to S3 such as `s3cmd` and `s4cmd` and FUSE filesystems such as s3fs and s3ql. There are also things like `rclone` which probably solve your entire problem for you. What are you currently using to talk to S3? — derobert, Jan 23 '17 at 22:49
@derobert i'm using the `aws cli` - If you have an example to help please feel free to answer the question. — Patrick B., Jan 23 '17 at 23:12
I'd think `rclone copy /data/files whatever:` would do everything for you... But anyway, you should [edit] your question to clarify which software you're using to talk to AWS. And if you're open to switching. — derobert, Jan 23 '17 at 23:13

score 16 · Answer 1 · edited Feb 20 '19 at 13:07

16

If you do aws s3 ls on the actual filename. If the filename exists, the exit code will be 0 and the filename will be displayed, otherwise, the exit code will not be 0:

aws s3 ls s3://bucket/filname
if [[ $? -ne 0 ]]; then
  echo "File does not exist"
fi

edited Feb 20 '19 at 13:07

Mr Chow

103
3

answered Jun 13 '17 at 17:55

onetwopunch

261
2
4

4

The problem with this is that `s3 ls` will list the file and give a return code of 0 (success) even if you provide a partial path. For example, `aws s3 ls s3://bucket/filen` will list the file `s3://bucket/filename`. – Donnie Cameron Mar 02 '19 at 02:02

score 3 · Answer 2 · edited Sep 07 '18 at 22:57

3

first answer is close but in cases where you use -e in shebang, the script will fail which you would most like not want. It is better to use wordcount. So you can use the below command:

wordcount=`aws s3 ls s3://${S3_BUCKET_NAME}/${folder}/|grep $${file}|wc -c`
echo wordcount=${wordcount}
if [[ "${wordcount}" -eq 0 ]]; then
do something
else
do something
fi

edited Sep 07 '18 at 22:57

Rui F Ribeiro

55,929
26
146
227

answered Aug 23 '18 at 20:33

Anuj Agrawal

31
1

1

`$${file}` will expand to the PID and `{file}`, probably not what you meant. – Benjamin W. Aug 23 '18 at 20:55

score 3 · Answer 3 · answered Mar 24 '20 at 06:18

3

Try the following :

aws s3api head-object --bucket ${S3_BUCKET} --key ${S3_KEY}

It retrieves the metadata of the object without retrieving the object itself. READ(s3:GetObject) access is required. .

answered Mar 24 '20 at 06:18

Subrata Das

171
4

This is great, especially since it returns a JSON if you want to find a specific field, it's easy to grab the value. – Alexis Wilke Jun 09 '20 at 21:49

score 0 · Accepted Answer · edited Jan 29 '17 at 21:20

0

I was able to do it using rclone[1] as @derobert has suggested.

The command is very simple:

rclone check sourcepath remote:s3bucketname

Example:

Let's imagine you want to check if the S3 bucket (bucket name: tmp_data_test_bucket) has all the files that this directory has: /tmp/data/

Command:

rclone check /tmp/data/ remote:tmp_data_test_bucket

[1] http://rclone.org/

edited Jan 29 '17 at 21:20

Jeff Schaller

66,199
35
114
250

answered Jan 24 '17 at 19:01

Patrick B.

119
1
1
5

1

rclone is not a native call, I don't consider this the best solution. – Mike Q Aug 20 '19 at 14:05

score 0 · Answer 5 · answered Aug 20 '19 at 15:44

I created two functions as an example because I figured I might want to know the size of the file and I may want to know if the file exists.

This function gets the size of the file and "returns" it as an echo:

s3_file_size() {
    if command -v aws &> /dev/null; then
        echo "$(aws s3 ls "${1}" --summarize | grep "Total.*Size" | grep -o -E '[0-9]+')"
        return 0
    else
        echo "Warn-${FUNCNAME[0]}, AWS command missing."
        return 1
    fi
}

This function uses the other one to determine if it receives a file size of 0 which will imply the file is essentially not there. (yes it will treat a file of size 0 as not there)

s3_does_file_exist() {
    if command -v aws &> /dev/null; then
        [[ $(s3_file_size "${1}") -lt 1 ]] && return 1 || return 0
    else
        echo "Warn-${FUNCNAME[0]}, AWS command missing."
        return 1
    fi
}

Check if file exists in S3 Bucket

5 Answers5