everyone!
I have a trouble with executing .sh script from browser.
echo command works well. But when I store the value in variable and pass it to another script as an argument it does not work.
Appreciate any help.
Example:
Here I pass $XX to another script
if [ -z "$QUERY_STRING" ]; then
exit 0
else
XX=`echo "$QUERY_STRING" | sed -n 's/^.*val_x=\([^&]*\).*$/\1/p' | sed "s/%20/ /g"`
if [ -z "$XX" ]; then
echo "X is empty"
else
echo "Processing..."
/sas/oljas_scripts/find_usage_of_tables.sh $XX
echo "Done"
fi
YY=`echo "$QUERY_STRING" | sed -n 's/^.*val_y=\([^&]*\).*$/\1/p' | sed "s/%20/ /g"`
ZZ=`echo "$QUERY_STRING" | sed -n 's/^.*val_z=\([^&]*\).*$/\1/p' | sed "s/%20/ /g"`
echo "val_x: " $XX
echo '<br>'
echo "val_y: " $YY
echo '<br>'
echo "val_z: " $ZZ
fi
The file you are trying to execute, /sas/oljas_scripts/find_usage_of_tables.sh, is not executable for the web server. Currently, it is:
-rwxrw-r-- 1 sas sas 1540 Nov 12 12:58 find_usage_of_tables.sh
Which means sas user can do everything, sas group can read and write but not execute, and everyone else can just read. Notably, the web server needs to be able to both read and execute the file, and it is failing to do so.
Either set the file to be readable and executable to all (not recommended):
chmod 755 /sas/oljas_scripts/find_usage_of_tables.sh
or set the group of the file to be a group the web server is in, and set the group permissions of the file accordingly:
chmod 754 /sas/oljas_scripts/find_usage_of_tables.sh
chgrp www-data /sas/oljas_scripts/find_usage_of_tables.sh
You may need sudo for the latter. Also, the web server's group name might not be www-data - inspect your web server configuration, or read it off of process list like this answer suggests.
Related
I have a config file that has details like
#pem_file username ip destination
./test.pem ec2-user 00.00.00.11 /Desktop/new/
./test1.pem ec2-user 00.00.00.22 /Desktop/new/
Now I need to know how can I fix the below script to get all the details using scp
while read "$(cat $conf | awk '{split($0,array,"\n")} END{print array[]}')"; do
scp -i array[1] array[2]#array[3]:/home/ubuntu/documents/xyz.xml array[4]
done
please help me.
Build your while read like this:
#!/bin/bash
while read -r file user ip destination
do
echo $file
echo $user
echo $ip
echo $destination
echo ""
done < <(grep -Ev "^#" "$conffile")
Use these variables to build your scp command.
The grep is to remove commented out lines.
If you prefer using an array, you can do this:
#!/bin/bash
while read -a line
do
echo ${line[0]}
echo ${line[1]}
echo ${line[2]}
echo ${line[3]}
echo ""
done < <(grep -Ev "^#" "$conffile")
See https://mywiki.wooledge.org/BashFAQ/001 for looping on files and commands output using while.
i am stuck with my piece of code any help is appreciated. This is the piece of code i am executing from jenkins.
#!/bin/bash
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[Tags[?Key==`Name`].Value|[0],PrivateIpAddress]' --output text | column | grep devtools > devtools
ip=`awk '{print $2}' devtool`
echo $ip
ssh ubuntu#$ip -n "aws s3 cp s3://bucket/$userlistlocation . --region eu-central-1"
cd builds/${BUILD_NUMBER}/
scp * ubuntu#$ip:/home/ubuntu
if [ $port_type == "normal" ]; then
if [ $duplicate_value == "no" ]; then
if [ $userlist == "uuid" ]; then
ssh ubuntu#$ip -n "export thread_size='"'$thread_size'"'; filename=$(echo $userlistlocation | sed -E 's/.*\/(.*)$/\1/') ; echo $filename ; echo filename='"'$filename'"'; chmod +x uuidwithduplicate.sh; ./uuidwithduplicate.sh"
fi
fi
fi
fi
userlistlocation --> is an user input it can be in any format /rahul/december/file.csv or simply it can be file.csv.
Through sed command i am able to get the output and stored in "filename" variable.
But when i try to echo $filename it's printing as echo $filename it should print as file.csv
this file.csv will be the source file for one more script to run i.e for uuidwithduplicate.sh
both userlistlocation and thread_size are specified through Jenkins job parameters.
I am not facing issues while exporting thread_size, only issue is with filename.
It's just printing echo $filename --> it should print file.csv
Breaking down the ssh command:
ssh ubuntu#$ip -n "export thread_size='"'$thread_size'"'; filename=$(echo $userlistlocation | sed -E 's/.*\/(.*)$/\1/') ; echo $filename ; echo filename='"'$filename'"'; chmod +x uuidwithduplicate.sh; ./uuidwithduplicate.sh"
Into segments of single/double quoted items
"export thread_size='"
'$thread_size'
"#'; filename=$(echo $userlistlocation | sed -E 's/./(.)$/\1/') ; echo $filename ; echo filename='#"
'$filename'
"'; chmod +x uuidwithduplicate.sh; ./uuidwithduplicate.sh"
Note: On the 3rd token, an '#' was added between double quotes and single quote to make it more readable. Not part of the command.
On surface few issues:
The '$thread_size' should be "$thread_size" to enable expansion
The 'echo $filename' is in double quote, resulting in expansion on the local host, where as setting filename=$(echo ...) is executed on the remote host.
There are two echo for filename, not sure why
Proposed solution is to move the setting of filename to the local host (simplify command), and move the thread_size into double quotes. It is possible to put complete command into single double-quoted item:
filename=$(echo $userlistlocation | sed -E 's/.*\/(.*)$/\1/')
ssh localhost -n "export thread_size='$thread_size'; echo '$filename' ; echo filename='$filename'; chmod +x uuidwithduplicate.sh; ./uuidwithduplicate.sh"
This script looks for all users that have the string RECHERCHE inside them. I tried running it in sudo and it worked, but then stopped at line 8 (permission denied). Even when removing the sudo from the script, this issue still happens.
#!/bin/bash
#challenge : user search and permission rewriting
echo -n "Enter string to search : "
read RECHERCHE
echo $(cat /etc/passwd | grep "/home" | cut -d: -f5 | grep -i "$RECHERCHE" | sed s/,//g)
echo "Changing permissions"
export RECHERCHE
sudo ./challenge2 $(/etc/passwd) &
The second script then changes permissions of each file belonging to each user that RECHERCHE found, in the background. If you could help me figure out what this isn't doing right, it would be of great service. I
#!/bin/bash
while read line
do
if [-z "$(grep "/home" | cut -d: -f5 | grep -i "$RECHERCHE")" ]
then
user=$(cut -f: -f1)
file=$(find / -user $(user))
if [$(stat -c %a file) >= 700]
then
chmod 700 file 2>> /home/$(user)/challenge.log
fi
if [$(stat -c %a file) < 600]
then
chmod 600 file 2>> /home/$(user)/challenge.log
fi
umask 177 2>> /home/$(user)/challenge.log
fi
done
I have to idea what I'm doing.
the $(...) syntax means command substitution, that is: it will be replaced by the output of the command within the paranthesis.
since /etc/passwd is no command but just a text-file, you cannot execute it.
so if you want to pass the contents of /etc/passwd to your script, you would just call it:
./challenge2 < /etc/passwd
or, if you need special permissions to read the file, something like
sudo cat /etc/passwd | ./challenge2
also in your challenge2 script, you are using $(user) which is wrong as you really only want to expand the user variable: use curly braces for this, like ${user}
/etc/passwd?
not what you were asking, but you probably should not read /etc/passwd directly anyhow.
if you want to get a list of users, use the following command:
$ getent passwd
this will probably give you more users than those stored in /etc/passwd, as your system might use other PAM backends (ldap,...)
I'm using scp command to copy file from one Linux host to another.
I run scp commend on host1 and copy file from host1 to host2. File is quite big and it takes for some time to copy it.
On host2 file appears immediately as soon as copying was started. I can do everything with this file even if copying is still in progress.
Is there any reliable way to find out if copying was finished or not on host2?
Off the top of my head, you could do something like:
touch tinyfile
scp bigfile tinyfile user#host:
Then when tinyfile appears you know that the transfer of bigfile is complete.
As pointed out in the comments, this assumes that scp will copy the files one by one, in the order specified. If you don't trust it, you could do them one by one explicitly:
scp bigfile user#host:
scp tinyfile user#host:
The disadvantage of this approach is that you would potentially have to authenticate twice. If this were an issue you could use something like ssh-agent.
On sending side (host1) use script like this:
#!/bin/bash
echo 'starting transfer'
scp FILE USER#DST_SERVER:DST_PATH
OUT=$?
if [ $OUT = 0 ]; then
echo 'transfer successful'
touch successful
scp successful USER#DST_SERVER:DST_PATH
else
echo 'transfer faild'
fi
On receiving side (host2) make script like this:
#!/bin/bash
SLEEP_TIME=30
MAX_CNT=10
CNT=0
while [[ ! -e successful && $CNT < $MAX_CNT ]]; do
((CNT++))
sleep($SLEEP_TIME);
done;
if [[ -e successful ]]; then
echo 'successful'
rm successful
# do somethning with FILE
fi
With CNT and MAX_CNT you disable endless loop (in case file successful isn't transferred).
Product MAX_CNT and SLEEP_TIME should be equal or greater expected transfer time. In my example expected transfer time is less than 300 seconds.
A checksum (md5sum, sha256sum ,sha512sum) of the local and remote files would tell you if they're identical.
For the situation where you don't have SSH access to the remote system - like an FTP server - you can download the file after it's uploaded and compare the checksums. I do this for files I send from production scripts at work. Below is a snippet from the script in which I do this.
MD5SRC=$(md5sum $LOCALFILE | cut -c 1-32)
MD5TESTFILE=$(mktemp -p /ramdisk)
curl \
-o $MD5TESTFILE \
-sS \
-u $FTPUSER:$FTPPASS \
ftp://$FTPHOST/$REMOTEFILE
MD5DST=$(md5sum $MD5TESTFILE | cut -c 1-32)
if [ "$MD5SRC" == "$MD5DST" ]
then
echo "+Local and Remote files match!"
else
echo "-Local and Remote files don't match"
fi
if you use inotify-tools,
then the solution will looks like this:
while ! inotifywait -e close $(dirname ${bigfile_fullname}) 2>/dev/null | \
grep -Eo "CLOSE $(basename ${bigfile_fullname})$">/dev/null
do true
done
echo "File ${bigfile_fullname} closed"
After some investigation, and discussion of the problem on other forums I have found one more solution. Maybe it can help somebody.
There is a command "lsof". It lists open files. During copying the file will be opened, so the command
lsof | grep filename
will return non empty result.
So you might want to make a while loop to wait until lsof returns nothing and proceed with your task.
Example:
# provide your file name here
f=<nameOfYourFile>
lsofresult=`lsof | grep $f | wc -l`
while [ $lsofresult != 0 ]; do
echo still copying file $f...
sleep 5
lsofresult=`lsof | grep $f | wc -l`
done; echo copying file $f is finished: `ls $f`
For the duplicate question, How to check if file has been scp 100% to the remote location , which was for an expect script, to know if a file is transferred completely, we can add expect 100% .. .. i.e something like this ...
expect -c "
set timeout 1
spawn scp user#$REMOTE_IP:/tmp/my.file user#$HOST_IP:/home/.
expect yes/no { send yes\r ; exp_continue }
expect password: { send $SCP_PASSWORD\r }
expect 100%
sleep 1
exit
"
if [ -f "/home/my.file" ]; then
echo "Success"
fi
If avoiding a second SSH handshake is important, you can use something like the following:
ssh host cat \> bigfile \&\& touch complete < bigfile
Then wait for the "complete" file to get created on the remote end.
I have a simple php page that writes a file to my server.
// open new file
$filename = "$name.txt";
$fh = fopen($filename, "w");
fwrite($fh, "$name".";"."$abbreviation".";"."$uid".";");
fclose($fh);
I then have a cron job that I know runs as root as test that and need that.
if [[ $EUID -ne 0 ]]; then
echo "This script must be run as root" 1>&2
exit 1
fi
The cronjob is a bash script that can detect the file exists, but it can't seem to read the contents of the file.
#!/bin/bash
######################################################
#### Loop through the files and generate coincode ####
######################################################
for file in /home/test/customcoincode/queue/*
do
echo $file
chmod 777 $file
echo "read file"
while read -r coinfile; do
echo $coinfile
echo "Assign variables from file"
#############################################
#### Set the variables to from the file #####
#############################################
coinName=$(echo $coinfile | cut -f1 -d\;)
coinNameAbreviation=$(echo $coinfile | cut -f2 -d\;)
UId=$(echo $coinfile | cut -f3 -d\;)
done < $file
echo "`date +%H:%M:%S` - $coinName : Your Kryptocoin is being compiled!"
echo $file
echo "copy $coinName file to generated directory"
cp -b $file /home/test/customcoincode/generatedCoins/$coinName.txt
echo "`date +%H:%M:%S` : Delete queue file"
# rm -f $file
done
echo $file recognises the file exists
echo $coinfile is blank
Yet when I nano ./coinfile.txt in terminal I can see clearly there is text in there
I run ls -l and I see that the file has the permissions
-rw-r--r-- 1 www-data www-data
I was under the impression that this would still mean the file can be read by other users?
Do I need to be able to execute the file if i am opening it and reading the contents?
Any advice would be greatly appreciated. I can expand and show my code if you want, but it was working before when I called a bash script to write the file... and that time it would save the file under root user with rwx for most and then could be read. But this then caused other issues in the php page, so is not an option.
You have:
while read -r coinfile; do
...
I see no indication that you're reading from $file. The command
read -r coinfile
will simply read from standard input (the -r merely affects the treatment of backslashes). In a cron job, if I recall correctly, standard input is empty or unavailable, which would explain why $coinfile is empty.
If you actually do read from $file -- for example, if your real code looks something like:
while read -r coinfile; do
...
done <$file
then you need to show us your entire script, or at least a self-contained version of it that exhibits the problem. Actually, you need to show us your entire script whether that's the problem or not.
http://sscce.org/