Is there a way to add in a parameter to aws s3 cp command to output the time taken to copy a file up to s3 or copy down a file to localhost? Can it output something like TimeTaken: 17 secs after the task is completed?
you can just use the bash time command:
time aws s3 ls
Related
I'm trying to run multiple shell commands through Docker using AWS Batch and boto3. When I try to submit multiple commands using the & sign as follows, the job fails.
My attempt
import boto3
client = boto3.client("batch")
response = client.submit_job(
jobName='AndrewJob',
jobQueue='AndrewJobQueue',
jobDefinition='AndrewJobDefinition',
containerOverrides={
'command': 'ls & python myjob.py'.split(),
},
timeout = {'attemptDurationSeconds': 100}
)
print(response)
The error is:
ls: cannot access '&': No such file or directory
According to the Docker Docs here https://docs.docker.com/engine/reference/builder/#cmd and this post here docker run <IMAGE> <MULTIPLE COMMANDS> it seems like this should be possible in shell form.
It appears that Batch is behaving like subprocess.Popen in that it executes the command as one command where the first argument is the command name and the remaining arguments are the command arguments. I got this to work with subprocess.Popen, so I bet it would work with Batch:
subprocess.Popen(["/bin/bash", "-c", "ls && echo \"Hello world\""])
I am trying to build a userdata load sequence which is constructed of several external files:
$ aws s3 cp s3://my-bucket/init.sh - | bash
$ echo "Some other custom commands"
$ aws s3 cp s3://my-bucket/more-stuff.sh - | bash
Now in init.sh there are some core functions that I need to use and they are not available in the other script sections since each one is a different bash session.
Is there are way to execute all these scripts and commands in one single bash session?
you should download the scripts and then run with source <filename>. Then all defined variables and functions are available for the other scripts.
$ aws s3 cp s3://my-bucket/init.sh ~/s3_init.sh
$ chomd 750 ~/s3_init.sh
$ source ~/s3_init.sh
...
For cp-options in aws s3 see https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html
I built a series of bash scripts to run BigQuery jobs for a data pipeline. These scripts are saved in a google cloud storage bucket. I pipe them to sh using this:
gsutil cat gs://[bucket]/[filename] | sh
Basically there is no problem if I run this from command line, but once I try running this command from within a bash script, I keep getting file not found errors?
It doesn't seem like a PATH issue (I may be mistaken) as calling $PATH from within the script shows where gsutil is located.
Is this a permissions issue?
I'm running this from within google cloud console shell in my browser. Any help is appreciated.
To start, try printing out the output (both stdout and stderr) of the gsutil cat command, rather than piping it to sh. If you're receiving errors from that command, this will help shed some light on why sh is complaining. (In the future, please try to copy/paste the exact error messages you're receiving when posting a question.)
Comparing the output of gsutil version -l from both invocations will also be helpful. If this is an auth-based problem, you'll probably see different values for the config path(s) lines. If this is the case, it's likely that either:
You're running the script as a different user than who you normally run gsutil as. gcloud looks under $HOME/.config/gcloud/... for credentials to pass along to gsutil... e.g. if you've run gcloud auth login as Bob, but you're running the script as root, gcloud will try looking for root's credentials instead of Bob's.
In your script, you're invoking gsutil directly from .../google-cloud-sdk/platform/gsutil/, rather than its wrapper .../google-cloud-sdk/bin/gsutil which is responsible for passing gcloud credentials along.
I am new to SSH & cronjobs so probably I am doing something wrong.
I am using a Google Cloud Engine for hosting my SSH Linux instance and I want to delete a snapshot by its name with a cronjob (I have also a create snapshot cronjob that works fine).
So I wrote this script:
1 * * * * sudo gcloud compute snapshots delete my-snapshot-name -q
This script should delete the snapshot every 1 minute (the 1 minute is just in order to see the result immediately after I will see it works, I will change it accordingly).
The snapshot is not deleted.
If I run the same script, not in the cronjob then it deletes it:
sudo gcloud compute snapshots delete my-snapshot-name -q
Some more details about how I create the cronjob:
In the google cloud SSH I run crontab -e.
I write the cron script written above.
Click on Ctrl + X.
It asks if I want to save modified buffer - I click Y.
It offers to write the file name to /tmp/crontab.xxxxxx/crontab, I click enter and the cronjob is created.
What am I doing wrong? What could be the cause that the delete does not work?
It's better to add this to root crontab than using sudo.
Not so familiar with GCE, however a few things to try,
1) Use the complete path to the gcloud binary
2) Check /var/log/syslog for 'CRON' and check what the error is.
I have am using a backup script that uses rsync to create a backup of my data, encrypts it and uploads it to the cloud. The script runs in an isolated docker container which only has access to the data it should backup (it cannot communicate with the host).
As the services whose data is backed up may not be running while backing up, I am currently executing the backup task (simplified) like this
#!/bin/bash
/etc/init.d/myservice stop
# the scripts blocks until the backup job is done
/opt/backup_data >> /var/log/backup.log
/etc/init.d/myservice start
and the output of the backup service looks like
backup started at 15-02-09 22:00:01
rsync started
/foo/bar
deleted /baz
[...]
rsync completed
starting upload
upload completed
backup completed
While this works, I would prefer to restart my services as soon as the rsync operation is finished, to keep the downtime as small as possible. But as all I have is the content of the log, how can I realized this?
TL;DR I run a command that outputs some text to stdout, which I redirect to file. I want to enforce that another command is executed as soon as the stdout of the first one contains a special string. If it doesn't (e.g. while something went wrong), I want to execute it anyway - but only once.
As the docker container's host can see the log file, I would use tail -f to "follow" the log file and pipe its output into awk. When awk "sees" that the rsync job is complete, it can restart the services:
tail -f backup.log | awk '/^rsync complete/ {system("/etc/init.d/myservice start")}'