I built a series of bash scripts to run BigQuery jobs for a data pipeline. These scripts are saved in a google cloud storage bucket. I pipe them to sh using this:
gsutil cat gs://[bucket]/[filename] | sh
Basically there is no problem if I run this from command line, but once I try running this command from within a bash script, I keep getting file not found errors?
It doesn't seem like a PATH issue (I may be mistaken) as calling $PATH from within the script shows where gsutil is located.
Is this a permissions issue?
I'm running this from within google cloud console shell in my browser. Any help is appreciated.
To start, try printing out the output (both stdout and stderr) of the gsutil cat command, rather than piping it to sh. If you're receiving errors from that command, this will help shed some light on why sh is complaining. (In the future, please try to copy/paste the exact error messages you're receiving when posting a question.)
Comparing the output of gsutil version -l from both invocations will also be helpful. If this is an auth-based problem, you'll probably see different values for the config path(s) lines. If this is the case, it's likely that either:
You're running the script as a different user than who you normally run gsutil as. gcloud looks under $HOME/.config/gcloud/... for credentials to pass along to gsutil... e.g. if you've run gcloud auth login as Bob, but you're running the script as root, gcloud will try looking for root's credentials instead of Bob's.
In your script, you're invoking gsutil directly from .../google-cloud-sdk/platform/gsutil/, rather than its wrapper .../google-cloud-sdk/bin/gsutil which is responsible for passing gcloud credentials along.
Related
I have thousands of gsutil commands in a shell script with the syntax
“gsutil cp gs://name of bucket/from-path to-path”
When I execute this script in my windows local using a gitbash it works fine. But when I am running this script in a Linux, it shows that the files are getting copied but when I look in the folder it is empty. Please help me.
Files should get saved in the destination.
The actual commands are:
gsutil cp gs://mycompany/archive/data/raw/integration/THY/2022/12/19/TKT_20221217.zip /home/ds102e/workspace/mft/
gsutil cp gs://mycompany/archive/data/raw/integration/SLK/2022/12/19/SQ9V_20221608.zip /home/ds102e/workspace/mft/
Is it possible you're on a Linux environment where gsutil is defined as an alias that runs gsutil in an isolated filesystem (e.g. within a docker container)? You can verify this by running type gsutil from your linux shell. As an example, see this post.
I am trying to find a way to record every single command that is executed by any user on the system.
Things that I have came across earlier.
It is possible to view shell commands executed from the terminal using ~/.bashrc_history file.
There is a catch here, It logs only those commands which were executed interactively from bash shell/terminal.
This solves one of my problems. But in addition to it, I would like to log those commands also which were executed as a part of the shell script.
Note: I don't have control over shell script. Therefore, adding verbose mode like #!/bin/bash -xe is not possible.
However, this can be assumed that I have root access as a system administrator.
Eg: I have another user that has access to the system. And he runs the following shell script using from his account.
#!/bin/sh
nmap google.com
and run as "$ sh script.sh"
Now, What I want is "nmap google.com" command should be logged somewhere once this file is executed.
Thanks in advance. Even a small help is appreciated.
Edit: I would like to clarify that users are unaware that they are being monitored. So I need a solution something at system level(may be agent running with root). I cannot depend on user to log suspicious activity. Of-course everyone will avoid such tricks to put blame on someone else if they do something fishy or wrong
I am aware that you were asking for Bash and Shell scripting and tagged your question accordingly, but in respect to your requirements
Record every single command that is executed by any user on the system
Users are unaware that they are being monitored
A solution something at system level
I am under the assumption that you are looking for Audit Logging.
So you may take advantage from articles like
Log all commands run by Admins on production servers
Log every command executed by a User
You can run the script in this way:
execute bash (it will override the shebang)
ts to prefix every lines
logs both in terminal and files
bash -x script.sh |& ts | tee -a /tmp/$(date +%F).log
You may ask the other user to create an alias.
Edit:
You may also add this into /etc/profile (sourced when users login)
exec > >(tee -a /tmp/$(date +%F).log)
Do it also for error output if needed. Keep it splited.
I have modified my rc.local script to run some python script at startup.
This python script seems to be started successfully.
As the script is running forever (intended) and I want to see what the script does, my question is:
Is there a way to access the shell that runs this script?
Yes, to see what is going on, I could log to some file, but what if that script needs to get input from the user via console?
Thanks for your help!
You will not be able to interact with the script run by rc.local. But you can see what it does by logging its output into dedicated files:
python myscript.py > /home/myhome/log/myscript.log 2> /home/myhome/log/myscript.err
where error messages go into a separate log file.
Note that your script will be executed by root, having permissions and ownership accordingly.
Here's a link to an earlier answer about this with a method to log all outputs of rc.local.
Now you can see in your log file, if the execution stops due to the script demanding input or indeed crashing, and then you can fix the script accordingly.
If you don't want to mess with rc.local for testing, you could also first run it through crontab on your or root's account (scheduled execution by user, see man crontab). This might be easier for debugging, and you can start it through rc.local once it works as you want.
I've been banging my head at this for some time and can't find an answer.
Disclaimer: I'm a Linux newbie.
So, I received a script to run on a VM at startup and I'm using an Azure template deployment and script extension so I can only run one command, and I run the command like so
sudo sh script.sh param1 param2 param3
I can change the way I run it as long as it remains one line.
My script is imported to the VM by Azure and run by Azure so I'm not sure of the inner workings but It kept failing because of several permission issues(I did not write the script, was just handed it by the guy who used to run it manually - and I have no idea what the actual issue with the permissions is) and I finally got it to run to completion with using heredoc in this manner:
#!/usr/bin/env bash
sudo su <<"EOF"
var1=$1
var2=$2
var3=$3
rest of script
EOF
Now, the script DOES work permission wise but the parameters aren't going through to the su environment - so I found out that I can remove the quotations and then it would work. but THEN I can't create new variables inside the heredoc part.
Can someone come to my aid? :) I have no idea how to solve this apart from re-writing the whole script which will probably take me ages.
Thank you!
I'm trying to upload files to Google Cloud Storage using cron in Linux, but it fails.
I have also set path and configuration in my script file as :
PATH=/bin/gsutil/
export BOTO_CONFIG="/home/ashu/.boto"
# rest of script
But still nothing works.
And if you are using default installation provided by Google Cloud - Compute Engine, most probably gsutil is in /snap/bin
PATH=$PATH:/snap/bin
It's a bit safer to do
PATH="$PATH":/bin/gsutil/
so you don't kill access to the usual places like /bin and /usr/bin and etc. You may not use them directly, but scripts you call might!
update: #ComputerDruid rightly points out that quotes keep spaces from causing trouble.
I had the same issue on Ubuntu 20.04 and I found that the simplest solution was to create a softlink between my gsutil installation and the system bin folder like this:
sudo ln -s /snap/bin/gsutil /usr/bin/gsutil
Apart from modifying the PATH, as suggested by pjz, did you try to look at the actual output from gsutil / cron?
Which reason is given for the commend failing? In case you need to catch the output of gsutil,
you can redirect standard output and error (stdout and stderr) to a file, and save it there.
E.g. if you're using Bash, you could redirect the output to gsutil_log.txt by modifying your crontab as:
*/1 * * * * /mypath/myscript.sh >> $HOME/gsutil_log.txt
This will redirect stdout and stderr and append any output to gsutil_log.txt in $HOME for myscript.sh that is called every minute by cron.
If the output is helpful, that should advance you a bit with debugging.
I removed my pip install and used following link for installation :
https://cloud.google.com/storage/docs/gsutil_install#specifications.
Also use of sudo should be avoided for path and export as it can lead to some issues.
PATH=$PATH:/root/gsutil/
export BOTO_CONFIG="/root/.boto"
# rest of script
Above code works well.
you can try using the full path gsutil command to use in crontab
/root/gcloud/gsutil cp ...