The below script works fine. But when I try to add a command to remote copy and then assign the variable FILENAME with the file received from the remote copy, the while loop doesn't work. I am quite new to scripting so I'm not able to find out what I'm missing. Please help!
#!/bin/sh
#SCRIPT: File processing
#PURPOSE: Process a file line by line with redirected while-read loop.
SSID=$1
ASID=$2
##rcp server0:/oracle/v11//dbs/${SSID}_ora_dir.lst /users/global/rahul/${ASID}_clone_dir.lst
##FILENAME=/users/global/rahul/${ASID}_clone_dir.lst
count=0
while read LINE
do
echo $LINE | sed -e "s/${SSID}/${ASID}/g"
count=`expr $count + 1`
done < $FILENAME
echo -e "\nTotal $count Lines read"
grep -v -e "pattern3" -e "pattern5" -e "pattern6" -e "pattern7" -e "pattern8" -e "pattern9" -e "pattern10" -e "pattern11" -e "
pattern12" ${ASID}_.lst > test_remote.test
When you say, "the while loop doesn't work", if you get an error message you should include that in your question to give us a clue.
Are you sure the rcp command is successful? The file /users/global/rahul/${ASID}_clone_dir.lst exists after the rcp is completed?
Btw your while loop is inefficient. This should be equivalent:
sed -e "s/${SSID}/${ASID}/g" < "$FILENAME"
count=$(wc -l "$FILENAME" | awk '{print $1}')
echo -e "\nTotal $count Lines read"
Related
I have a config file that has details like
#pem_file username ip destination
./test.pem ec2-user 00.00.00.11 /Desktop/new/
./test1.pem ec2-user 00.00.00.22 /Desktop/new/
Now I need to know how can I fix the below script to get all the details using scp
while read "$(cat $conf | awk '{split($0,array,"\n")} END{print array[]}')"; do
scp -i array[1] array[2]#array[3]:/home/ubuntu/documents/xyz.xml array[4]
done
please help me.
Build your while read like this:
#!/bin/bash
while read -r file user ip destination
do
echo $file
echo $user
echo $ip
echo $destination
echo ""
done < <(grep -Ev "^#" "$conffile")
Use these variables to build your scp command.
The grep is to remove commented out lines.
If you prefer using an array, you can do this:
#!/bin/bash
while read -a line
do
echo ${line[0]}
echo ${line[1]}
echo ${line[2]}
echo ${line[3]}
echo ""
done < <(grep -Ev "^#" "$conffile")
See https://mywiki.wooledge.org/BashFAQ/001 for looping on files and commands output using while.
I have a working bash script to create backups and upload them as a tar archive to a remote sftp server.
After the upload, the script should remove all but the latest 20 backup files. I can't use any, pipe, grep, whatever on the sftp. Also I don't get the file-listing result handled in my bash-script.
export SSHPASS=$(cat /etc/backup/pw)
SFTPCONNECTION=$(cat /etc/backup/sftp-connection)
sshpass -e sftp $SFTPCONNECTION - << SOMEDELIMITER
ls -lt backup-*.tar
quit
SOMEDELIMITER
There is this nice oneliner, but I did not figure out how the use it in my case (sftp).
This script deletes all tar files in the given directory except the last 20 ones. The -t flag sorts by time & date. The <<< redirect expands $RESULT feed's it into the stdin of the while loop. I'm not entirely pleased with it as it has to create multiple connections, but with sftp I don't believe there is another way.
RESULT=`echo "ls -t path/to/old_backups/" | sftp -i ~/.ssh/your_ssh_key user#server.com | grep tar`
i=0
max=20
while read -r line; do
(( i++ ))
if (( i > max )); then
echo "DELETE $i...$line"
echo "rm $line" | sftp -i ~/.ssh/your_ssh_key user#server.com
fi
done <<< "$RESULT"
Thanks to codelitt I went with this solution:
export SSHPASS=$(cat /etc/backup/pw)
SFTPCONNECTION="username#host"
RESULT=`echo "ls -tl backup*.tar" | sshpass -e sftp $SFTPCONNECTION | grep -oP "backup.*\.tar" `
i=0
max=24
while read -r line; do
# echo "$line "
(( i++ ))
if (( i > max )); then
echo "DELETE $i...$line"
echo "rm $line" | sshpass -e sftp $SFTPCONNECTION
fi
done <<< "$RESULT"
It's a slight modification of his version:
it counts/removes only files named backup*.tar
it uses ls -l (for line based listings)
I had to use sshpass instead of a certificate-based authentication. The sftp password is inside /etc/backup/pw
I want to search for patterns in a file and remove the lines containing the pattern. To do this, am using:
originalLogFile='sample.log'
outputFile='3.txt'
temp=$originalLogFile
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp > $outputFile
temp=$outputFile
done <$whiteListOfErrors
This works fine for the first iteration. For the second run, it throws :
grep: input file ‘3.txt’ is also the output
Any solutions or alternate methods?
The following should be equivalent
grep -v -f "$whiteListOfErrors" "$originalLogFile" > "$outputFile"
originalLogFile='sample.log'
outputFile='3.txt'
tmpfile='tmp.txt'
temp=$originalLogFile
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp > $outputFile
cp $outputfile $tmpfile
temp=$tmpfile
done <$whiteListOfErrors
Use sed for this:
sed '/.*pattern.*/d' file
If you have multiple patterns you may use the -e option
sed -e '/.*pattern1.*/d' -e '/.*pattern2.*/d' file
If you have GNU sed (typical on Linux) the -i option is comfortable as it can modify the original file instead of writing to a new file. (But handle with care, in order to not overwrite your original)
Used this to fix the problem:
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp | tee $outputFile
temp=$outputFile
done <$falseFailures
Trivial solution might be to work with alternating files; e.g.
idx=0
while ...
let next='(idx+1) % 2'
grep ... $file.$idx > $file.$next
idx=$next
A more elegant might be the creation of one large grep command
args=( )
while read line; do args=( "${args[#]}" -v "$line" ); done < $whiteList
grep "${args[#]}" $origFile
I have a problem with my bash script. I want to excluding from processing files that are listed in the exclude.log. After a file is processed it is written in to the exclude log.
for I in `ls $1 | grep ./exclude.log -v`
do
echo "Procesing ...."
echo $I >> ./exclude.log
done
$I is not assigned a value.
Also your grep is not right formulated.
You possibly want
LIST=$( grep -v -f /path/to/exclude.log * )
for I in $LIST
do
echo "Procesing ...."
echo $I >> /path/to/exclude.log
done
Make sure you don't have any empty lines in exclude.log
You can use this while loop:
while read -r l; do
echo "$l";
done < <(fgrep -v -wf exclude.log <(printf "%s\n" "$1"/*))
I am looking for a linux command that searches a string in a text file,
and highlights (colors) it on every occurence in the file, WITHOUT omitting text lines (like grep does).
I wrote this handy little script. It could probably be expanded to handle args better
#!/bin/bash
if [ "$1" == "" ]; then
echo "Usage: hl PATTERN [FILE]..."
elif [ "$2" == "" ]; then
grep -E --color "$1|$" /dev/stdin
else
grep -E --color "$1|$" $2
fi
it's useful for stuff like highlighting users running processes:
ps -ef | hl "alice|bob"
Try
tail -f yourfile.log | egrep --color 'DEBUG|'
where DEBUG is the text you want to highlight.
command | grep -iz -e "keyword1" -e "keyword2" (ignore -e switch if just searching for a single word, -i for ignore case, -z for treating as a single file)
Alternatively,while reading files
grep -iz -e "keyword1" -e "keyword2" 'filename'
OR
command | grep -A 99999 -B 99999 -i -e "keyword1" "keyword2" (ignore -e switch if just searching for a single word, -i for ignore case,-A and -B for no of lines before/after the keyword to be displayed)
Alternatively,while reading files
grep -A 99999 -B 99999 -i -e "keyword1" "keyword2" 'filename'
command ack with --passthru switch:
ack --passthru pattern path/to/file
I take it you meant "without omitting text lines" (instead of emitting)...
I know of no such command, but you can use a script such as this (this one is a simple solution that takes the filename (without spaces) as the first argument and the search string (also without spaces) as the second):
#!/usr/bin/env bash
ifs_store=$IFS;
IFS=$'\n';
for line in $(cat $1);
do if [ $(echo $line | grep -c $2) -eq 0 ]; then
echo $line;
else
echo $line | grep --color=always $2;
fi
done
IFS=$ifs_store
save as, for instance colorcat.sh, set permissions appropriately (to be able to execute it) and call it as
colorcat.sh filename searchstring
I had a requirement like this recently and hacked up a small program to do exactly this. Link
Usage: ./highlight test.txt '^foo' 'bar$'
Note that this is very rough, but could be made into a general tool with some polishing.
Using dwdiff, output differences with colors and line numbers.
echo "Hello world # $(date)" > file1.txt
echo "Hello world # $(date)" > file2.txt
dwdiff -c -C 0 -L file1.txt file2.txt