In my files commands are like :
File1.sh:
log_file='20210101'
echo -e "\n Output is \n" >> $log_file 2>&1
File2.sh
echo "Something"
log_file='20210102'
hive -e -f my_file_name.hql >> $log_file 2>&1
File3.sh
cat myfile
echo "something"
ehco "new line"
log_file='2021-01-01_23_59_59'
hive -e "Select * from tables' >> $log_file 2>&1
I am looking for a command to remove the line where defined variable 'log_file' from all the files
AND
remove only '>> $log_file 2>&1' part from all the files
Output in files:
File1.sh:
echo -e "\n Output is \n"
File2.sh
echo "Something"
hive -e -f my_file_name.hql
File3.sh
cat myfile
echo "something"
ehco "new line"
hive -e "Select * from tables'
I tried multiple ways with sed but nothing worked, few of them are :
I tried :
sed -e 's/*[>> $]log_file//g' test.sh
sed -e 's/*[>> $]log_file[ 2>&1]//g' test.sh
sed -e 's/\>\> \$log_file \2\>$//' test.sh
sed -e 's/">> $log_file 2>&1"/""/g' test.sh
This sed command is a two-step process, and makes a backup of all files (just in case); on my machine (and with the 3 samples you posted) it does the right thing:
sed -i.bak -e '/^log_file=/d' -e 's/>> \$log_file 2>&1//g' File*
Related
I have the following script file that writes files to s3 from a local file system:
#!/bin/bash
CURR_DIR=`dirname $0`
SCRIPT_NAME="$(basename $0)"
LOG_FILE=$(echo $SCRIPT_NAME | cut -f 1 -d '.')
TODAY=$(date '+%Y-%m-%d')
NOW=$(date -d "$(date +%Y-%m-%d)" +%Y"-"%m"-"%d)
LOG_PATH="$CURR_DIR"/logs/"$LOG_FILE"-$TODAY.log
LOG="[$(date '+%Y-%m-%d %H:%M:%S,%3N')] INFO {$LOG_FILE} -"
ERROR_LOG="[$(date '+%Y-%m-%d %H:%M:%S,%3N')] ERROR {$LOG_FILE} -"
BUCKET="s3.bucket.example"
OUT_FOLDER="path/to/folderA"
S3_PUSH="s3://$BUCKET/$OUT_FOLDER"
exec &>> $LOG_PATH
echo "$LOG Copying files to local out folder..." >> $LOG_PATH
cp /path/to/folderA/*.* /path/to/folderB
echo "$LOG Command returned code:" $?
if [ "$(ls -A path/to/folderA/)" ]; then
FILES="$(ls path/to/folderA/*)"
for file in $FILES ; do
echo "$LOG File $file found for sync" >> $LOG_PATH
echo "$LOG Pushing $file to S3 /Folder..." >> $LOG_PATH
echo -n "$LOG " ; s3cmd put -c /home/config/.s3cfg "$file" "$S3_PUSH"/
echo "$LOG Command returned code:" $?
echo "$LOG Copying $file to local backup..." >> $LOG_PATH
mv "$file" /path/to/folderA/backup/
echo "$LOG Command returned code:" $? >> $LOG_PATH
RCC=$?
if [ $? -eq 0 ]
then
echo "$LOG Command returned code:" $?
else
echo "$ERROR_LOG Command returned code:" $?
fi
done
else
echo "$LOG No files found for sync." >> $LOG_PATH
fi
And the output is coming out in a specific grok pattern needed for me to parse this output as logs into Elastic Search, however the line 27 output is as follows:
[2021-09-02 08:15:25,629] INFO {TestGrokScriptPattern} - upload: '/path/to/folderA/File.txt' -> 's3://s3.bucket.example/Path/To/Bucket/File.txt' [1 of 1]
0 of 0 0% in 0s 0.00 B/s done
that upload and 0 of 0 0%... Line is created by the exec & command executed on line 16.
How can I get that output to not go to the next line without the date, time and script name preceeding it in order to not break the log pattern I am trying to create?
Rather than redirect output on each line, you can wrap the body of the script in a single block and then handle the output of the entire block in one place. You can then process that output with the stream editor sed. For example:
if true; then # Always true. Just simplifies redirection.
echo "Doing something..."
command_with_output
command_with_more_output
echo "Done."
fi | sed "s/^/${LOG}/" > ${LOG_PATH} 2>&1
The sed expression means: Substitute (s) the beginning of each line (^) with the contents of the LOG variable.
Using 2>&1 at the end also eliminates the need for the exec &>> $LOG_PATH command.
I created a shell script to loop through some files in a directory, unzip them, add in a date field, zip them back up, and then move them to the hadoop file system. But, when I run the script, it seems to go right to the next line without waiting for gunzip to complete. How do I tell it to wait for it to complete before moving to the next line?
FILENAME="/datatst/toproc/*"
for i in $FILENAME
do
echo "file name is: " $i
FILENAMEv2=$(basename "${i}" .gz )
echo "Stripped file name is: " $FILENAMEv2
DATEPART=$(echo $i| cut -d"." -f1| cut -d"-" -f2-)
echo "Datepart is: " $DATEPART
FileDir="/datatst/unzip/$FILENAMEv2"
echo "unzip directory file is: " $FileDir
echo "unzipping file..."
gunzip $i > -c $FileDir && trash $i
echo "unzipping done..."
echo "sed operation begin..."
sed -i 's/^/'$DATEPART' /g' $FileDir
echo "sed operation done..."
echo "zip operation begin..."
gzip $FileDir -c > /datatst/tomove/$i && trash $FileDir
su hadoop fs -put /datatst/tomove/$i /user/hdfs/
done
I think you should try changing ...
gunzip $i > -c $FileDir && trash $i
... to ...
gunzip $i -c > $FileDir && trash $i
... I am not sure what the trash is.
Updated as per comment from user*
I am not getting the use of -f from this statement in this program.
if [ -f $file ]
I got if but why we use -f here?
Here is whole program:
case $ch in
1)
echo -e "\n This is File Create Operation"
echo -e "\n Please Enter File Name:"
read file
if [ -f $file ]
then
echo -e "\n File already existing!!!"
else
touch $file
echo -e "\n File Created!!!"
echo -e "\n Created File can be Checked From Following Table"
ls -t|head -n 15
fi
;;
[ is usually both an alias to test and a shell builtin. [ something ] is equivalent to test something
See help [ in bash to learn the bash builtin's version, and man test to learn about the non-builtin binary /bin/test.
you'll see that [ -f file ] is true if file exist and is a regular file, and false otherwise (ie it returns 0 only if file is a regular file)
I have this
#! /bin/bash
cd ~
hostname=`hostname`
cat /opt/ip.txt | while read line;
do
# do something with $line here
RES=`ping -c 2 -q $line | grep "packet loss"`
echo "---" >> /opt/os-$hostname.txt
echo "---"
echo "$line $RES" >> /opt/os-$hostname.txt
echo "$line $RES"
done
How I can make the script multi-threaded? I would like to speed up the performance.
You can use the <(...) notation for starting a subprocess and then cat all the outputs together:
myping() {
ping -c 2 -q "$1" | grep "packet loss"
}
cat <(myping hostname1) <(myping hostname2) ...
To use a loop for this, you will need to build the command first:
cat /opt/ip.txt | {
command='cat'
while read line
do
command="$command "'<'"(myping $line)"
done
eval "$command"
}
If you really want the delimiting --- of your original, I propose to add an echo "---" in the myping.
If you want to append the output to a file as well, use tee:
eval "$command" | tee -a /opt/os-$hostname.txt
DELETED.
WAS UN USEFUL ? NO THREAD IN BASH.
I need suggestions on how to improve the below script for better efficiency ( both time and resource, and for better error detection). Here is a simple description on what the script does:
Functionality: The script runs in crontab for "ins" user very minute, searches for a .DAT file in "input path". If it finds the .dat file, moves it to "working directory", executes it as a batch, and after completion of the execution of .dat file moves it to "output folder". The .dat file contains a series a similar commands to insert numbers into database.
#!/usr/bin/ksh
def_path="/apps/ins/"
env_path="/apps/ins/eir/bin"
input_path="/apps/ins/eir/batch/input/"
work_path="/apps/ins/eir/batch/working/"
output_path="/apps/ins/eir/batch/output/"
moved_path="/apps/ins/eir/batch/processed/"
log="/apps/ins/BLA/log/"
date=`date '+%d%b%y'`
cd $input_path
listcount=`ls -rt *.dat |wc -l`
list=`ls -rt *.dat`
echo "`date +%H:%M:%S`| Total No of DAT files available are # $listcount #\nName of the files are...\n $list " >> $log/$date.log 2>&1
if [[ -e $def_path/.bla_processing ]];
then
echo "`date +%H:%M:%S`| Script is already running" >> $log/$date.log 2>&1
exit
fi
for fname in `ls *.dat | awk -F. '{print $1}'`
do
touch $def_path/.bla_processing
mv $input_path/$fname.dat $work_path/$fname.dat
echo "##################################################" >> $log/$date.log 2>&1
echo "## Filename = $fname.dat ## Time = `date +%H:%M:%S` ##" >> $log/$date.log 2>&1
echo "##################################################" >> $log/$date.log 2>&1
cd $env_path
. /apps/ins/.profile >> $log/$date.log 2>&1
echo "Username is `whoami`" >> $log/$date.log 2>&1
$env_path/mip_cmd EXECUTE:$work_path/$fname.dat,$output_path/$fname.out,100; >> $log/$date.log 2>&1
sleep 2
echo "`date +%H:%M:%S`| Moving the file *** $fname.dat *** to path |$moved_path|" >> $log/$date.log 2>&1
mv $work_path/$fname.dat $moved_path/$fname.dat.moved
cmd_exec=`cat $output_path/$fname.out |grep ":" |wc -l`
echo "`date +%H:%M:%S`| Total commands executed are `expr $cmd_exec / 2`" >> $log/$date.log 2>&1
echo "`date +%H:%M:%S`| Total Sucessfully executed commands are `cat $output_path/$fname.out |grep "C1:00000;" |wc -l`" >> $log/$date.log 2>&1
echo "--------------------------------------------------" >> $log/$date.log 2>&1
echo "#### SCRIPT WILL PROCESS THE NEXT FILE ###########" >> $log/$date.log 2>&1
echo "--------------------------------------------------" >> $log/$date.log 2>&1
echo "" >> $log/$date.log 2>&1
rm $def_path/.bla_processing
exit
done
Since you are using "$def_path/.bla_processing" as a form of lock, you should probably check its existense first thing before you proceed with the rest of the script.
Also, "touch $def_path/.bla_processing" and "rm $def_path/.bla_processing" could be moved out of the for loop.
What your code is doing now is, only the first file is processed, and the script will exit. So the call to "exit" in the end of the for loop is not necessary.
As an example (after applying the above suggestions):
touch $def_path/.bla_processing
for fname in `ls *.dat | awk -F. '{print $1}'`
....
# remove the call to exit
done
rm $def_path/.bla_processing