Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [amazon-s3]

41 questions
11
votes
5 answers

Check if file exists in S3 Bucket

This directory /data/files/ has thousands files like: 1test 2test 3test [...] 60000test 60001test I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be offline and because of that the file is…
Patrick B.
  • 119
  • 1
  • 1
  • 5
8
votes
3 answers

How can we generate the session token in aws

i have aws access key and secret key with me. i wanted session token to be updated in aws credential file (~/.aws/credentials), how will i get it? I want them to be generated in command line.
Vandhana
  • 313
  • 1
  • 3
  • 10
8
votes
3 answers

Is it possible to get aws account id with only aws access key and secret key in command line (CLI)

Is it possible to get aws account id with only aws access key and secret key in command line (CLI) I have access key and secret key with me. Is it possible to get the account id using those in command line.
Vandhana
  • 313
  • 1
  • 3
  • 10
8
votes
3 answers

Search inside s3 bucket with logs

How to search for a string inside a lot of .gz files in Amazon S3 bucket subfolder? I tried to mount it via s3fs and zgrep but it's sooooo slow. Do you use any other methods? Maybe is there any Amazon service which I could use to quickly zgrep them?
Michal_Szulc
  • 256
  • 1
  • 3
  • 13
5
votes
1 answer

Duplicity + Amazon S3 Glacier. How Much "Unfreezing" to Resume Backup

It's been a couple of months since I last did a Duplicity backup [to Amazon S3] and, in the meantime thanks to an automatic 30-day rule I have on my Amazon buckets, my S3 backup has been moved from the 'normal' Reduced Redundancy storage to Amazon…
stíobhart
  • 51
  • 1
  • 4
5
votes
2 answers

Unable to pass a bash variable as a python argument in bash

For some reason I cannot pass the bash variable $FOLDER as a python argument on the following code. The python script downloads some files from amazon s3. Bash script: #!/bin/bash FOLDER=$(./aws get $BUCKET"/location.txt") python…
Bris
  • 61
  • 1
  • 1
  • 5
4
votes
1 answer

xargs with aws cli (amazon s3 copy)

I am trying to speed up the process of copying hundreds of file between 2 S3 buckets in my account. Bucket1 dt=2017-01-01/ file1, file2, file3 ... file100 .. .. dt=2017-01-31/ Each file is a ~300-400 MB and even 1 GB in some cases. Now, I need…
Abi
  • 143
  • 1
  • 5
3
votes
2 answers

Get the size of a S3 Bucket's sub-folder through bash script

I am trying to write a bash script to get the total size of sub folders in a S3 bucket. My bucketpath s3://path1/path2/subfolders Inside the path2 folder i have many sub-folder like 2019_06 2019_07 2019_08 2019_09 2019_10 2019_11 2019_12 I need to…
clarie
  • 85
  • 2
  • 11
3
votes
2 answers

How to sort s3cmd ls output by date?

I want to sort directories in an s3 storage by the date embedded in their name. When I run s3cmd ls s3://xyz/private/backups/mails/daily/ | awk '{print $2}' it lists the directories like…
Pankaj Jha
  • 133
  • 6
3
votes
1 answer

egrep in bash script not working

I have this set of commands : s3ls="aws s3 ls $ssyss3/ --recursive --human-readable --summarize" egrepCommand="egrep '$currentDate|Total'" totalSize=$(echo $s3ls|$egrepCommand| awk -F 'Total Size:' '{print $2}'|sed '/^$/d') echo $totalSize but I'm…
user352882
  • 31
  • 2
3
votes
1 answer

Find files then move them and rename them at the same time?

Currently I use three steps to gzip some static assets and then use s3cmd to an S3 bucket (technically it's a Digital Ocean Spaces bucket). Here's what I do: $ find . -type f -name '*.css' | xargs -I{} gzip -k -9 {} $ find . -type f -name…
cfx
  • 133
  • 5
3
votes
1 answer

How to get private IP address of EC2 after spinning the AWS from local machine/ at Jenkins outside AWS

Assume we already spin the AWS using cloudformation plugin from Jenkins outside AWS and now, how to get private IP address after spinning the AWS at my local machine/jenkins using any API methods? I tried ruby aws-sdk, REST API methods to get the…
user176867
  • 31
  • 3
2
votes
1 answer

s3fs complains about SSH key or SSL cert - how to fix?

I downloaded and installed s3fs 1.73 on my Debian Wheezy system. The specific steps I took were, all as root: apt-get -u install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support ./configure…
user
  • 28,161
  • 13
  • 75
  • 138
2
votes
3 answers

autofs : dynamic mounting rule for s3 bucket

I successfully implement autofs to automatically mount s3 bucket to the server which running ubuntu server 14.04.5 by following this tutorials. but the number of bucket (that needs to automatically mounted) is dynamic, means its can be increase or…
kahidna
  • 21
  • 4
1
vote
0 answers

How to make a directory public and private in a single bucket

I am using amazon S3 bucket to store my data in two different directories in single bucket, I want that my one directory should be globally accessible and another directory should be private. Is there any policy which I can use ?
Nitesh B.
  • 563
  • 2
  • 7
  • 20
1
2 3