I was just wondering what command i need to put into the terminal to read a text file, eliminate all lines that do not contain a certain keyword, and then print those lines onto a new file. for example, the keyword is "system". I want to be able to print all lines that contain system onto a new separate file. Thanks
grep is your friend.
For example, you can do:
grep system <filename> > systemlines.out
man grep and you can get additional useful info as well (ex: line numbers, 1+ lines prior, 1+lines after, negation - ie: all lines that do not contain grep, etc...)
If you are running Windows, you can either install cygwin or you can find a win32 binary for grep as well.
grep '\<system\>'
Will search for lines that contain the word system, and not system as a substring.
below grep command will solve ur problem
grep -i yourword filename1 > filename2
with -i for case insensitiveness
without -i for case sensitiveness
to learn how grep works on ur server ,refer to man page on ur server by the following command
man grep
grep "system" filename > new-filename
You might want to make it a bit cleverer to not include lines with words like "dysystemic", but it's a good place to start.
Related
I need to find a string in a textfile, delete the line containing it, and save the file. The string is found (read from) another textfile, containing hundreds of different strings, one per row. The process is to go on from the first to the last string in the file.
Any (hopefully easy to use) text editors (on Windows OS) recommended ? To achive the task.
I am not into serious day-to-day editing. So I'd be ever so happy if the task could be accomplished with a easy-to-use but still reliable editor.
Thanks a bunch,
Frank
You can try notepad++ since it has a lot of plugins, also a great search algorithm. I did a similar task where I had to do a lot of search/replace stuff, and used a plugin I dug up from the internet, can't remember the name exactly (try google-ing I think it's replaacc for notepad++ or something similar).
On unix/linux/cygwin:
grep -v -f pattern_file unmodified_file > new_file
Remove all lines containing the patterns in pattern_file from unmodified_file, write to new_file.
grep -v outputs lines not matching any pattern. -f reads patterns from a file.
On windows this appears equivalent to running this at the command prompt:
FINDSTR /V /G:pattern_file unmodified_file > new_file
That's it. If you already have the two source files, it's a one-liner.
pattern_file is going to be whitespace and case sensitive unless you delve into other options, which are described with FINDSTR /?
Using sed:
sed -n '/PATTERN/n;p' FILE > FILE.new # then copy FILE.new to FILE
Tells sed to not output anything by default (-n), find the pattern (/PATTERN/) and skip this line if found (n), otherwise print the line (;p). If you have GNU sed you do can just call
sed -i -n '/PATTERN/n;p' FILE`
which automatically updates the file due to (-i /--in-place).
I am quite new to shell scripting.
I am scraping a website and the scraped text contains a lot of repetitions. Usually they are the menus on a forum, for example. Mostly, I do this in Python, but I thought that sed command will save me reading and printing the input, loops etc. I want to delete thousands of repeated lines from the same single file. I do not want to copy it to another file, because I will end up with 100 new files. The following is a shadow script which I run from the bash shell.
#!/bin/sed -f
sed -i '/^how$/d' input_file.txt
sed -i '/^is test$/d' input_file.txt
sed -i '/^repeated text/d' input_file.txt
This is the content of the input file:
how to do this task
why it is not working
this is test
Stackoverflow is a very helpful community of programmers
that is test
this is text
repeated text is common
this is repeated text of the above line
Then I run in the shell the following command:
sed -f scriptFile input_file.txt
I get the following error
sed: scriptFile line 2: untermindated `s' command
How can I correct the script, and what is the correct syntax of the command I should use to get it work?
Any help is highly appreciated.
assuming you know what your script is doing, it's very easy to put them into a script. in your case, the script should be:
/^how$/d
/^is test$/d
/^repeated text/d
that's good enough.
to make the script alone to be executable is easy too:
#!/usr/bin/env sed -f
/^how$/d
/^is test$/d
/^repeated text/d
then
chmod +x your_sed_script
./your_sed_script <old >new
here is a very good and compact tutorial. you can learn a lot from it.
following is an example from the site, just in case the link is dead:
If you have a large number of sed commands, you can put them into a file and use
sed -f sedscript <old >new
where sedscript could look like this:
# sed comment - This script changes lower case vowels to upper case
s/a/A/g
s/e/E/g
s/i/I/g
s/o/O/g
s/u/U/g
Wouldn't it be easier to do it with egrep followed by a mv, for example
egrep -v 'pattern1|pattern2|pattern3|...' <input_file.txt >tmpfile.txt
mv tmpfile.txt input_file.txt
Each pattern would describe the lines being deleted, much like in sed. You would not end up with additional files, because the mv removes them.
If you have so many pattern, that you don't want to specify them directly on the command line, you can store them in a file use the -f option of egrep.
I am trying to find the number of lines that contain the word. I realize this is possible with the wc command and the -l flag, but I'm not 100% sure how to do it. Any ideas?
Instead of wc, you can use grep like this:
grep -c "word" file
I am a windows user having basic idea about LINUX and i encountered this command:
cat countryInfo.txt | grep -v "^#" >countryInfo-n.txt
After some research i found that cat is for concatenation and grep is for regular exp search (don't know if i am right) but what will the above command result in (since both are combined together) ?
Thanks in Advance.
EDIT: I am asking this as i dont have linux installed. Else, i could test it.
Short answer: it removes all lines starting with a # and stores the result in countryInfo-n.txt.
Long explanation:
cat countryInfo.txt reads the file countryInfo.txt and streams its content to standard output.
| connects the output of the left command with the input of the right command (so the right command can read what the left command prints).
grep -v "^#" returns all lines that do not (-v) match the regex ^# (which means: line starts with #).
Finally, >countryInfo-n.txt stores the output of grep into the specified file.
It will remove all lines starting with # and put the output in countryInfo-n.txt
This command would result in removing lines starting with # from the file countryInfo.txt and place the output in the file countryInfo-n.txt.
This command could also have been written as
grep -v "^#" countryInfo.txt > countryInfo-n.txt
See Useless Use of Cat.
Hello
I'm looking some script or program that use keywords or pattern search in files ex. php, html, etc and show where is this file
I use command cat /home/* | grep "keyword"
but i have too many folders and files and this command causes big uptime :/
I need this script to find fake websites (paypal, ebay, etc)
find /home -exec grep -s "keyword" {} \; -print
You don't really say what OS (and shell) you are using. You might want to retag your question to help us out.
Because you mention cat | ... , I am assuming you are using a Unix/Linux variant, so here are some pointers for looking at files. (bmargulies solution is good too).
I'm looking some script or program that use keywords or pattern search in files
grep is the basic program for searching files for text strings. Its usage is
grep [-options] 'search target' file1 file2 .... filen
(Note that 'search target' contains a space, if you don't surround spaces in your searchTarget with double or single quotes, you will have a minor error to debug.)
(Also note that 'search target' can use a wide range of wild-card characters, like .,?,+,,., and many more, that is beyond the scope of your question). ... anyway ...
As I guess you have discovered, you can only cram so many files at a time into the comand-line, even when using wild-card filename expansion. Unix/linux almost always have a utiltiyt that can help with that,
startDir=/home
find ${startDir} -print | xargs grep -l 'Search Target'
This, as one person will be happy to remind you, will require further enhancements if your filenames contain whitespace characters or newlines.
The options available for grep can vary wildly based on which OS you are using. If you're lucky, you type the following to get the man page for your local grep.
man grep
If you don't have your page buffer setup for a large size, you might need to do
man grep | page
so you can see the top of the 'document'. Press any key to advance to the next page and when you are at the end of the document, the last key press returns you to the command prompt.
Some options that most greps have that might be useful to you are
-i (ignore case)
-l (list filenames only (where txt is found)
There is also fgrep, which is usually interpretted to mean 'file' grep
becuase you can give it a file of search targets to scan for, and is used like
fgrep [-other_options] -f srchTargetsFile file1 file2 ... filen
I need this script to find fake websites (paypal, ebay, etc)
Final solution
you can make a srchFile like
paypal.fake.com
ebay.fake.com
etc.fake.com
and then combined with above, run the following
startDir=/home
find ${startDir} -print | xargs fgrep -il -f srchFile
Some greps require that the -fsrchFile be run together.
Now you are finding all files starting /home, searching with fgrep for paypay, ebay, etc in all files. The -l says it will ONLY print the filename where a match is found. You can remove the -l and then you will see the output of what is found, prepended with the filename.
IHTH.