5

For some reason I cannot pass the bash variable $FOLDER as a python argument on the following code. The python script downloads some files from amazon s3.

Bash script:

#!/bin/bash
FOLDER=$(./aws get $BUCKET"/location.txt")
python /data/pythonscript.py $FOLDER
#

The output of the $FOLDER is a regenerated date eg. 2014/07/31/14-16-34 which is used as a path.

Here is the python script: 

#!/usr/bin/python

import boto, sys
from boto.s3.connection import S3Connection

access_key = 'accesskey'
secret_key = 'secretkey'
bucket_name = 'a name'
folder_path =  str(sys.argv[1]) if len(sys.argv) > 1 else ''

print("Forwarded folder path " + folder_path)

conn = S3Connection(access_key, secret_key)
bucket = conn.get_bucket(bucket_name)

print("Bucket Location:" + bucket.get_location())

for key in bucket.list(prefix=folder_path, delimiter=''):
        if '.' in key.name:
                file_name = key.name[len(folder_path)+1:]
                print("Downloading file " + file_name)
                key.get_contents_to_filename('/data/temp/' + file_name)

When I execute the bash script without changing the python /data/pythonscript.py $FOLDER line, I get the following output: 

Forwarded folder path 2014/07/31/14-16-34 
Buckect Location:

But when I change it to python /data/pythonscript.py 2014/07/31/14-16-34 , everything works:

Forwarded folder path 2014/07/31/14-16-34
Bucket Location: 
Downloading 2014/07/31/14-16-34/FacetedSearch.zip 
Downloading file FacetedSearch.zip
Downloading 2014/07/31/14-16-34/Location.zip
Downloading file Location.zip
Downloading 2014/07/31/14-16-34/LocationPage.zip
Downloading file LocationPage.zip
drs
  • 5,363
  • 9
  • 40
  • 69
Bris
  • 61
  • 1
  • 1
  • 5
  • What about this doesn't work? Does the python script not get called with `$FOLDER` expanded? – drs Jul 31 '14 at 14:20
  • Python is getting the $FOLDER argument but rest of the code doesn't work as expected. But if I change the /data/pythonscript.py $FOLDER to /data/pythonscript.py 2014/07/31/14-16-34 in bash script, it works. Python file also works from command line independently. – Bris Jul 31 '14 at 14:29
  • You've framed your question nicer than a lot of new users, but "rest of the code doesn't work as expected" is still pretty vague. What happens? Is there an error? What is the value of `folder_path` when using the bash variable `$FOLDER` as the argument? – drs Jul 31 '14 at 14:42
  • Man, is `s3download.py` the same script, you called `pythonscript.py` earlier? – Boris Burkov Jul 31 '14 at 15:42
  • Just a minor complaint, but you mistyped bucket: `Buckect Location` in line 16. – polym Jul 31 '14 at 15:49
  • Yes Bob the script is called 'pythonscript.py' and not `s3download.py` , sorry for the confusion – Bris Jul 31 '14 at 16:06

2 Answers2

7

Perhaps the aws bash command is returning non-printable characters that you don't see with print(). Try removing them with tr:

FOLDER=$(./aws get $BUCKET"/location.txt" | tr -cd "[:print:]")
drs
  • 5,363
  • 9
  • 40
  • 69
3

Try to put $FOLDER inside double quotes:

python script.py "$FOLDER"

Change the first line of your script to

#!/usr/bin/env python

This should work.

And be more clear on your questions.

As @drs said, your question is still incomplete:

  • Your examples are still vague and with flaws, like what is the correct name of your script, is it pythonscript or s3download?
  • Is it "work as expected" supposed to have a blank value for bucket.get_location()?
Community
  • 1
Brandt
  • 282
  • 1
  • 2
  • 10
  • 1
    I tried your suggestions but unfortunately, I am still getting the same result. The script is named `pythonscript.py` and the `bucket.get_location()` indeed returns a blank value. But apart from that blank value the script manages to download the files from amazon s3 when I use the mentioned method. – Bris Jul 31 '14 at 16:04