I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC.
If the file is not there in machineB, then it should be there in machineC for sure.
I am using GNU Parallel here to download five files in parallel. And everything works fine if all the files are there -
#!/bin/bash
export PRIMARY=/data01/primary
export FILERS_LOCATION_1=machineB
export FILERS_LOCATION_2=machineC
export MEMORY_MAPPED_LOCATION=/bexbat/data/be_t1_snapshot
PRIMARY_PARTITION=(550 274 2 546 278 6 558 282 10 554 286 14) # this will have more file numbers
export dir1=/bexbat/data/re_t1_snapshot/20140501
# just iterating the file and doing ls and exit if any of the file is missing
for el in "${PRIMARY_PARTITION[#]}"
do
ssh david#$FILERS_LOCATION_1 ls $dir3/t1_weekly_1680_"$el"_200003_5.data || ssh david#$FILERS_LOCATION_2 ls $dir3/t1_weekly_1680_"$el"_200003_5.data || echo "File number $el missing on both the filers for primary partition." >&2; exit 1
done
echo "All files present. Starting to copy now."
# copy the files now
Problem Statement:-
Before copying any files, I want to see whether all the files are already present in either of the machines (machineB or machineC) or not. If any of the file is missing, then I need to print out which file is missing and exit out of the shell script with non zero status.
Above script is not working as the way I have described. If it sees any of the file is present, then it exits automatically, it's not moving in the for loop to look for other files. And I am not sure why?
Is there anything wrong I am doing?
ssh doesn't preserve quoting, so you need to escape commands locally to be unescaped by the remote shell.
for el in "${PRIMARY_PARTITION[#]}"
do
printf -v cmd '%q ' test -e "$dir3/t1_weekly_1680_${el}_200003_5.data"
ssh "david#$FILERS_LOCATION_1" "$cmd" \
|| ssh "david#$FILERS_LOCATION_2" "$cmd" \
|| { echo "File number $el missing on both the filers for primary partition." >&2;
exit 1; }
done
That ssh line in the loop doesn't do what you expect. The semi-colon has a lower precedence than the other operators, so when you suffix the line with ; exit 1 that will be executed always. You could just use an if statement:
if ! ssh david#$FILERS_LOCATION_1 ls $dir3/t1_weekly_1680_"$el"_200003_5.data && \
! ssh david#$FILERS_LOCATION_2 ls $dir3/t1_weekly_1680_"$el"_200003_5.data;
then
echo "File number $el missing on both the filers for primary partition." >&2
exit 1
fi
Related
This is my small bash script code and i want to print the number of files created in the directory :
#! /bin/sh
home_dir=/home/var/foo
Count= ls /$home_dir|wc -l
echo -e "$Count files are existed in the directory $home_dir"
exit 0
My expected output is :
9 files are existed in the directory /home/var/foo
but i got the below output:
9
files are existed in the directory /home/var/foo
can you help what went wrong in my above code? Also please suggest if this is the correct way to print the number of files in the directory
This works for me:
Count=$(ls /$xml_dir|wc -l)
To print on the same line
echo -ne "$Count files are existed in the directory $home_dir"
Add argument n to the echo.
I'm trying to run a bash script on Windows 7, using cygwin. The script takes two lists of file destinations (files are the same sprinkled in different pairs of folders), iterates through them and detects if the files changed.
#!/bin/bash
src=(
"./src/index.js"
"./src/index_2.js"
)
dest=(
"./client/src/index.js"
"./client/src/index_2.js"
)
arraylength=${#src[#]};
for (( i=0; i<${arraylength}; i++ ));
do
DIFF=$(diff -u ${src[$i]} ${dest[$i]})
if [ $? != 0 ]; then
echo "$DIFF"
echo "Files ${src[$i]} and ${dest[$i]} are not equal!"
exit 1
fi
done
echo "All files are equal"
When I run the command like ./shareddiff.sh, the command executes without errors, but displays nothing (no echo message). Even when I manualy change one of the index.js or index_2.js files - it doesn't detect the change.
Any idea what I could be doing wrong?
You are misusing diff in passing the file arguments; you can compare two files or two directories, not two list of files.
SYNOPSIS
diff [OPTION]... FILES
FILES are 'FILE1 FILE2' or 'DIR1 DIR2' or 'DIR FILE' or 'FILE DIR'. If
--from-file or --to-file is given, there are no restrictions on
FILE(s). If a FILE is '-', read standard input. Exit status is 0 if
inputs are the same, 1 if different, 2 if trouble.
try
diff -uR src client/src
I'm working on a bash script that pulls a file from an FTP site only if the timestamp on remote is different than local. After it puts the file, it copies the file over to 3 other computers via samba (smbclient).
Everything works, but the file copies even if the wget -N ftp://insertsitehere.com returns a value that the file on the remote was not newer. What would be the best way to check the output of the script so that the copy only happens if a new version was pulled from FTP?
Ideally, I'd like the copy to the computers to preserve the timestamp just like the wget -N command does, too.
Here is an example of what I have:
#!/bin/bash
OUTDIR=/cats/dogs
cd $OUTDIR
wget -N ftp://user:password#sitegoeshere.com/filename
if [ $? -eq 0 ]; then
HOSTS="server1 server2 server3"
for i in $HOSTS; do
echo "Uploading to $i..."
smbclient -A /root/.smbclient.authfile //$i/path -c "lcd /cats/dogs; put fiilename.txt"
if [ $? -eq 0 ]; then
echo "Upload to $i successful..."
else
echo "There was an issue uploading to host $i..."
fi
done
else
echo "There was an issue with the FTP Download...."
exit 1
fi
The return value of wget is different than 0 only if there is an error. If -N is in use and the remote file is older than the local file, it will still have a return value of 0, so you cannot use that to check if the file has been modified.
You could check the mtime of the file to see if it changed, or the content. For example, you could use something like:
md5_old=$( md5sum filename.txt 2>/dev/null )
wget -N ftp://user:password#sitegoeshere.com/filename.txt
md5_new=$( md5sum filename.txt )
if [ "$md5_old" != "$md5_new" ]; then
# Copy filename.txt to SMB servers
fi
Regarding smbclient, unfortunately there is no way to preserve timestamps in either get or put commands. If you need it, you must use some different tool (scp -p, rsync -t...)
touch -r foo.txt foo.old
wget -N example.com/foo.txt
if [ foo.txt -nt foo.old ]
then
echo 'Uploading to server1...'
fi
"Save" the current timestamp into a new empty file
Use wget --timestamping to only download the file if it is newer
If file is newer than the "save" file, do stuff
Below is my shell script in which I am trying to append $element in my below scp call in the if statement block.
for element in ${x[$key]}; do # no quotes here
printf "%s\t%s\n" "$key" "$element"
if [ $key -eq 0 ]
then
scp david#machineB:/data/be_t1_snapshot/20131215/t1_$element_5.data /data01/primary/.
fi
done
But whenever I am running my above shell script, I always get like this -
scp david#machineB:/data/be_t1_snapshot/20131215/t1_.data No such file or directory
When I have taken a close look in the above error message, the above scp statement is not right as it should be -
scp david#machineB:/data/be_t1_snapshot/20131215/t1_0_5.data /data01/primary/.
The value of $element should get replaced with 0 but somehow my above appending logic is not working. Is there anything wrong I am doing in the way I am appending $element in my above scp command
try t1_${element}_5.data
scp david#machineB:/data/be_t1_snapshot/20131215/t1_${element}_5.data /data01/primary/.
when you use t1_$element_5.data, bash will replace $element_5 with its value,
you don't have $element_5 defined, so you are getting
t1_.data No such file or directory
I'm creating a bash script that will run a process in the background, which creates a socket file. The socket file then needs to be chmod'd. The problem I'm having is that the socket file isn't being created before trying to chmod the file.
Example source:
#!/bin/bash
# first create folder that will hold socket file
mkdir /tmp/myproc
# now run process in background that generates the socket file
node ../main.js &
# finally chmod the thing
chmod /tmp/myproc/*.sock
How do I delay the execution of the chmod until after the socket file has been created?
The easiest way I know to do this is to busywait for the file to appear. Conveniently, ls returns non-zero when the file it is asked to list doesn't exist; so just loop on ls until it returns 0, and when it does you know you have at least one *.sock file to chmod.
#!/bin/sh
echo -n "Waiting for socket to open.."
( while [ ! $(ls /tmp/myproc/*.sock) ]; do
echo -n "."
sleep 2
done ) 2> /dev/null
echo ". Found"
If this is something you need to do more than once wrap it in a function, but otherwise as is should do what you need.
EDIT:
As pointed out in the comments, using ls like this is inferior to -e in the test, so the rewritten script below is to be preferred. (I have also corrected the shell invocation, as -n is not supported on all platforms in sh emulation mode.)
#!/bin/bash
echo -n "Waiting for socket to open.."
while [ ! -e /tmp/myproc/*.sock ]; do
echo -n "."
sleep 2
done
echo ". Found"
Test to see if the file exists before proceeding:
while [[ ! -e filename ]]
do
sleep 1
done
If you set your umask (try umask 0) you may not have to chmod at all. If you still don't get the right permissions check if node has options to change that.