Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am writing a bash script to check if an array element is in a file.
For example:
I have an array of errors errors=("1234" "5678" "9999")
I have a file that contains patterns of strings
123400 452612 9999A0 1010EB
I am looking to loop over the file that contains the errors and check to see if any of the array elements matches any string pattern in the file. If it does then give me back the exact array pattern it matched in the file for further processing.
Any ideas on how I can do this?
Here's a way where you only need to invoke grep once:
$ grep -oFf <(printf "%s\n" "${errors[#]}") file
1234
9999
The -f option is to specify a file that contains the pattern. I use a process substitution to "contain" the patterns, one per line.
The -F option specifies plain-text matching: I assume your "errors" array won't contain regular expressions.
Sounds like you just want a loop:
for error in "${errors[#]}"; do
if grep -qE "(^| )$error( |\$)" file; then
# $error was found in the file
fi
done
This matches the error preceded by the start of the line or a space, and followed by a space or the end of the line.
I made an effort to not match appearances of the errors within substrings but if you don't care, then you could change the grep command to this:
grep -qF "$error" file
This will return success if the error string occurs anywhere on the line.
The script goes like this,
#/bin/bash
errors=("1234" "5678" "9999")
for error in "${errors[#]}"
do
grep -o "$error" file
done
For a sample file,
$ cat file
123400 452612 9999A0 1010EB
The script produces an output
$ ./script.sh
1234
9999
meaning the above two keys from the array have matched in the file. The -o flag in grep is to identify only the matching parts from the array. An excerpt from the man grep page.
-o, --only-matching
Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I have a file word.txt
$ cat word.txt
cat
dog
rat
bird
I have a URL like this
https://example.com/?word=
I want to generate a URL list like this
https://example.com/?word=cat
https://example.com/?word=cat,dog
https://example.com/?word=cat,dog,rat
https://example.com/?word=cat,dog,rat,bird
I have 177 words so how can I automate this process with Bash or any other easy programming
Read the input line by line, add the line to the URL. Don't include the comma for the first line.
#! /bin/bash
url='https://example.com/?word='
while read -r line ; do
url+=$comma$line
comma=,
echo "$url"
done < word.txt
This task can be accomplished with a single GNU sed command:
sed -n 's|^|https://example.com/?word=|; :a; p; N; s/\n/,/; ba' word.txt
That should be more efficient than the plain bash.
Explanation:
-n With this option, sed only produces output when explicitly told to via the p command.
s|^|https://example.com/?word=| Replaces the beginning of the line with the https://example.com/?word=. This command effectively prepends that string to the pattern space.
:a Label a for branch command (b). Used when looping through the lines.
p Prints the pattern space.
N Adds a newline to the pattern space, then appends the next line of input to the pattern space.
s/\n/,/ Replaces the newline with the comma (,).
ba Jumps to the label a. This effectively creates a loop for all input lines except the first line.
Another variant:
url='https://example.com/?word='
while read word; do
words+=($word) list="${words[#]}"
printf '%s%s\n' "$url" ${list// /,}
done < word.txt
You can use something like this if you want to do it in python
words=["a","b","c"]
joinedWords = ",".join(words)
url="https://example.com/?word="
print(url,joinedWords)
reqUrl = url+joinedWords
print(reqUrl)
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
I have an hexadecimal string s and a file f, i need to search the first occurence of that string in the file and save that in a variable with his offset. I thought that the right way to do that is convert the file to hex and search that with a grep. The main problem is that i saw a lot of commands(hexdump,xxd,etc.) to convert but none of them actually work. Any suggestion?
My attempt was like this:
xxd -plain $f > $f
grep "$s" .
output should be like:
> offset:filename
A first approach without any error handling could look like
#!/bin/bash
BINFILE=$1
SEARCHSTRING=$2
HEXSTRING=$(xxd -p ${BINFILE} | tr -d "\n")
echo "${HEXSTRING}"
echo "Searching ${SEARCHSTRING}"
OFFSET=$(grep -aob ${SEARCHSTRING} <<< ${HEXSTRING} | cut -d ":" -f 1)
echo ${OFFSET}:${BINFILE}
I've used xxd here because of Does hexdump respect the endianness of its system?. Please take also note that according How to find a position of a character using grep? grep will return multiple matches, not only the first one. The offset will be counted beginning from 1, not 0. To substract 1 from the variable ${OFFSET} you may use $((${OFFSET}-1)).
I.e. search for the "string" ELF (HEX 454c46) in a system binary will look like
./searchHEX.sh /bin/yes 454c46
7f454c460201010000000000000000000...01000000000000000000000000000000
Searching 454c46
2:/bin/yes
I would use regex for this as well:
The text file:
$ cat tst.txt
1234567890x1fgg0x1cfffrr
A script you can easily change/extend yourself.
#! /bin/bash
part="$(perl -0pe 's/^((?:(?!0(x|X)[0-9a-fA-F]+).)*)(0(x|X)[0-9a-fA-F]+)(.|\n)*/\1:\3\n/g;' tst.txt)"
tmp=${part/:0x*/}
tmp=${#tmp}
echo ${part/*:0x/$tmp:0x} # Echoes 123456789:0x1f
Regex:
^((?:(?!0x[0-9a-fA-F]+).)*) = Search for the first entry that's a hexadecimal number and create a group of it (\1).
(0x[0-9a-fA-F]+) = Make a group of the hexadecimal number (\3).
(.|\n)* = Whatever follows.
Please note that tmp=${part/:0x*/} could cause problems if you have text like :0x before the hexadecimal number that is caught.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I want to search for a String by navigating to a particular line, How to do this in shell scripting?
For example,
I have
this is the first line
this is the Second line
This is the Third line
Now here i would want to look for string "Third" by going to 3rd line.
Any help is appreciated, Thank you.
Try stringing together cat, sed, and grep.
sed '3!d' filename | grep Third
The unnamed or anonymous pipe (|) and redirection (<, >) are powerful features of many shells. They allow one to combine a set of commands to perform a more complex function.
In the case of this question there were two clear steps,
1) Operate on a specific line of a file (e.g. filter a file)
2) Search the output of the filter for a specific string
Recognizing that there were two steps is a strong indicator that two commands will need to be combined. Therefore, the problem can be solved by finding a solution to each step and then combining them in to one command with pipes and redirection.
If you know about the Stream Editor (sed), it may come to your mind when thinking about how to accomplish the first step of filtering the file. If not searching for, "linux get a specific line of a file" this OS question comes up high in the search results.
$ cat tmp.txt
this is the first line
this is the Second line
This is the Third. line
$ sed '3!d' tmp.txt
This is the Third. line
Knowing that grep can be search for lines with the string of interest the next challenge is to figure out how to get the output of sed as the input to grep. The pipe (|) solves this problem.
sed '3!d' filename | grep Third
Example output:
$ sed '3!d' tmp.txt | grep Third
This is the Third. line
$
Another powerful concept in shell scripting is the exit status. The grep command will set the exit status to 0 when a match is found and 1 when a match is not found. The shell stores the exit status in a special variable named $? (for bash). Therefore, one could use the exit status to conditionally determine the next step in the shell script. The example below does not implement conditions (like if, else). The example below shows the exit status value using the echo command.
$ sed '3!d' tmp.txt | grep Third
This is the Third. line
$ echo $?
0
$ sed '3!d' tmp.txt | grep third
$ echo $?
1
$
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I want to search for a string (which I don't know unless I look inside the files) on Linux command-line.
Example:
A file 1 with text inside
A file 2 with text inside
In both files the word "Apple" is existent.
I want to echo this word (which exists in both files) in a file or store it in a variable.
How is this possible?
You can get a list of all the unique words in a file using:
grep -o -E '\w+' filename | sort -u
where -E '\w+' is matching words and -o outputs the matching parts. We can then use the join command which identifies matching lines in two files, along with process substitution to pass in the results of our word finder:
join <(grep -o -E '\w+' filename1 | sort -u) <(grep -o -E '\w+' filename2 | sort -u)
If there are no duplicates whiten a single file you could use cat file1 file2 |sort | uniq -d
$ cat input_one.txt
FIREFOX a
FIREFOX b
Firefox a
firefox b
$ cat input_two.txt
CHROME a
FIREFOX a
EXPLORER a
$ while read line; do grep "$line" input_two.txt ; done < input_one.txt
FIREFOX a
Explanation:
while will loop every line with input_two.txt file as input and will store the temporary line in the line variable.
In every line will search for it in the input_one.txt file and -o option will make to print only the matched part.
EDIT: See comments
You can write the script to handle this.
You need loop a word on file 1, in the loop use grep command (grep -nwr "" -e "$word") to find a word in file 2.
If match, echo a word.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Suppose I have a file with warnings. Each warning in a new line with an id that has 3 capital letters followed by 3 digits only, should be replaced by its id.
Example:
SIM_WARNING[ANA397]: Node q<159> for vector output signal does not exist
The output should be ANA397 and the rest of line is deleted.
How to do so using sed?
I don't think you need sed for that. A simple grep with --only-matching could do, as in:
grep -E 'SIM_WARNING\[(.)\]' --only-matching
should work for you.
Where:
-E does "enhanced regular expressions. I think we need those for capturing with ( )
then follows the pattern, which consists of the fixed SIM_WARNING, followed by a match inside the square brackets
--only-matching simply makes grep print only matching content
In other words: by using ( match ) you tell grep that you only care about the thing in that match pattern.
for id in $(grep -o "^SIM_WARNING\[[A-Z][A-Z][A-Z][0-9][0-9][0-9]\]" test1.bla | grep -o "[A-Z][A-Z][A-Z][0-9][0-9][0-9]" test1.bla ); do echo $id; done
This finds ANA397 from the below.
SIM_WARNING[ANA397]: Node q<159> for vector output signal does not exist
First of all, you have to choose how to use the IDs, for example if you need to cycle the file first or the IDs later...
E.G. (Cycle file first)
exec 3<file
while read -r line <&3; do
id="$(printf "%s" "${line}" | sed -e "s/.*\[\([[:alnum:]]\+\)\].*/\1/")"
### Do something with id
done
exec 3>&-
Otherwise you can decide to cycle the output of sed...
E.G.
for id in $(sed -e "s/.*\[\([[:alnum:]]\+\)\].*/\1/" file); do
### Do something with id
done
Both of the examples should work with posix shell (If I am not missing something...), but shell like posh may not support classes as [[:alnum:]], you can substitute them with the equivalent [a-zA-Z0-9], as every guide will teach you.
Note that the check is not on 3 letters and 3 digits, but for any letter and digit between brackets ([ and ]).
EDIT:
If your lines start with SIM_WARNING you can discriminates those lines with -e "/^SIM_WARNING/! d"
For a strict check on 3 letters and 3 digits you can use -e "s/.*\[\([a-zA-Z][a-zA-Z][a-zA-Z][0-9][0-9][0-9]\)\].*/\1/"
So taking the example above you can do somethin like:
for id in $(sed -e "/^SIM_WARNING/! d" -e "s/.*\[\([a-zA-Z][a-zA-Z][a-zA-Z][0-9][0-9][0-9]\)\].*/\1/" file)
### Do something with id
done