I am trying to understand the below command. Looked many tutorials and references. Can anyone help What actually it does.
sed -e '$d' -e '1,1d' File1 >File2
$d wil delete the last line
To delete lines between n and m use n,md. ie. 1,1d is same as 1d, which deletes the first line
So sed -e '$d' -e '1,1d' File1 >File2 will remove first and last line from File1 and send to file2
Related
I have two lists list1 and list2 with a filename on each line. I want a result with all filenames that are only in list2 and not in list1, regardless of specific file extensions (but not all). Using Linux bash, any commands that do not require any extra installations. In the example lists, I do know all file extensions that I wish to ignore. I made an attempt but it does not work at all, I don't know how to fix it. Apologies for my inexperience.
I wish to ignore the following extensions:
.x
.xy
.yx
.y
.jpg
list1.txt
text.x
example.xy
file.yx
data.y
edit
edit.jpg
list2.txt
text
rainbow.z
file
data.y
sunshine
edit.test.jpg
edit.random
result.txt
rainbow.z
sunshine
edit.test.jpg
edit.random
My try:
while read LINE
do
line2=$LINE
sed -i 's/\.x$//g' $LINE $line2
sed -i 's/\.xy$//g' $LINE $line2
sed -i 's/\.yx$//g' $LINE $line2
sed -i 's/\.y$//g' $LINE $line2
then sed -i -e '$line' result.txt;
fi
done < list2.txt
Edit: I forgot two requirements. The filenames can have . in them and not all filenames must have an extension. I know the extensions that must be ignored. I ammended the lists accordingly.
An awk solution might be more efficient for this task:
awk '
{ f=$0; sub(/\.(xy?|yx?|jpg)$/,"",f) }
NR==FNR { a[f]; next }
!(f in a)
' list1.txt list2.txt > result.txt
comm can do precisely this.
You can preprocess the input:
strip the suffices
sort (comm expects sorted input)
remove duplicates
ss()( sed 's/\.\(x\|xy\|yx\|y\|jpg\)$//' "$#" | sort -u )
comm -13 <(ss list1.txt) <(ss list2.txt) >result.txt
Your code was:
while read LINE
do
line2=$LINE
sed -i 's/\.x$//g' $LINE $line2
sed -i 's/\.xy$//g' $LINE $line2
sed -i 's/\.yx$//g' $LINE $line2
sed -i 's/\.y$//g' $LINE $line2
then sed -i -e '$line' result.txt;
fi
done < list2.txt
Some issues that immediately jump out:
syntax error - then/fi but no matching if
you never access list1
you don't quote variables when you use them, so whitespace and special characters will cause problems
while read ... sed ... sed ... sed ... is inefficient - multiple invocations of sed instead of just one, and a loop that sed would perform implicitly
sed expects file arguments not strings
sed -i will try to overwrite input file arguments
you use result.txt as both input and output to sed but never assign any contents to it
you try to use data ($line) as sed commands, instead of applying sed commands to that data
because you used single-quotes, sed -i -e '$line' will attempt to run a (non-existent) sed command line on the last line of input ($)
g option to s/// does nothing when search is anchored
I'd use join:
$ join -t. -j1 -v2 -o 2.1,2.2 <(sort list1.txt) <(sort list2.txt) | sed 's/\.$//'
rainbow.z
sunshine
(The bit of sed is needed to turn sunshine. into sunshine)
I've 95 files that looks like :
2019-10-29-18-00/dev/xx;512.00;0.4;/var/x/xx/xxx
2019-10-29-18-00/dev/xx;512.00;0.68;/xx
2019-10-29-18-00/dev/xx;512.00;1.84;/xx/xx/xx
2019-10-29-18-00/dev/xx;512.00;80.08;/opt/xx/x
2019-10-29-18-00/dev/xx;20480.00;83.44;/var/x/x
2019-10-29-18-00/dev/xx;3584.00;840.43;/var/xx/x
2019-10-30-00-00/dev/xx;2048.00;411.59;/
2019-10-30-00-00/dev/xx;7168.00;6168.09;/usr
2019-10-30-00-00/dev/xx;3072.00;1036.1;/var
2019-10-30-00-00/dev/xx;5120.00;348.72;/tmp
2019-10-30-00-00/dev/xx;20480.00;2033.19;/home
2019-10-30-12-00;/dev/xx;5120.00;348.72;/tmp
2019-10-30-12-00;/dev/hd1;20480.00;2037.62;/home
2019-10-30-12-00;/dev/xx;512.00;0.43;/xx
2019-10-30-12-00;/dev/xx;3584.00;794.39;/xx
2019-10-30-12-00;/dev/xx;512.00;0.4;/var/xx/xx/xx
2019-10-30-12-00;/dev/xx;512.00;0.68;/xx
2019-10-30-12-00;/dev/xx;512.00;1.84;/var/xx/xx
2019-10-30-12-00;/dev/xx;512.00;80.08;/opt/xx/x
2019-10-30-12-00;/dev/xx;20480.00;83.44;/var/xx/xx
2019-10-30-12-00;/dev/x;3584.00;840.43;/var/xx/xx
For some lines I've 2019-10-29-18-00/dev and for some other lines, I've 2019-10-30-12-00;/dev/
I want to add the ; before the /dev/ where it is missing, so for that I use this sed command :
sed 's/\/dev/\;\/dev/'
But How I can apply this command for each lines where the ; is missing ? I try this :
for i in $(cat /home/xxx/xxx/xxx/*.txt | grep -e "00/dev/")
do
sed 's/\/dev/\;\/dev/' $i > $i
done
But it doesn't work... Can you help me ?
Could you please try following with GNU awkif you are ok with it.
awk -i inplace '/00\/dev\//{gsub(/00\/dev\//,"/00;/dev/")} 1' *.txt
sed solution: Tested with GNU sed for few files and it worked fine.
sed -i.bak '/00\/dev/s/00\/dev/00\;\/dev/g' *.txt
This might work for you (GNU sed & parallel):
parallel -q sed -i 's#;*/dev#;/dev#' ::: *.txt
or if you prefer:
sed -i 's#;*/dev#;/dev#' *.txt
Ignore lines with ;/dev.
sed '/;\/dev/{p;d}; s^/dev^;/dev^'
The /;\/dev/ check if the line has ;/dev. If it has ;/dev do: p - print the current line and d - start from the beginning.
You can use any character with s command in sed. Also, there is no need in escaping \;, just ;.
How I can apply this command for each lines where the ; is missing ? I try this
Don't edit the same file redirecting to the same file $i > $i. Think about it. How can you re-write and read from the same file at the same time? You can't, the resulting file will be in most cases empty, as the > $i will "execute" first making the file empty, then sed $i will start running and it will read an empty file. Use a temporary file sed ... "$i" > temp.txt; mv temp.txt "$i" or use gnu extension -i sed option to edit in place.
What you want to do really is:
grep -l '00/dev/' /home/xxx/xxx/xxx/*.txt |
xargs -n1 sed -i '/;\/dev/{p;d}; s^/dev^;/dev^'
grep -l prints list of files that match the pattern, then xargs for each single one -n1 of the files executes sed which -i edits files in place.
grep for filtering can be eliminated in your case, we can accomplish the task with a single sed command:
for f in $(cat /home/xxx/xxx/xxx/*.txt)
do
[[ -f "$f" ]] && sed -Ei '/00\/dev/ s/([^;])(\/dev)/\1;\2/' "$f"
done
The easiest way would be to adjust your regex so that it's looking a bit wider than '/dev/', e.g.
sed -i -E 's|([0-9])/dev|\1;/dev|'
(note that I'm taking advantage of sed's flexible approach to delimiters on substitute. Also, -E changes the group syntax)
Alternatively, sed lets you filter which lines it handles:
sed -i '/[0-9]\/dev/ s/\/dev/;/dev/'
This uses the same substitution you already have but only applied on lines that match the filter regex
I have a file as below.
Hi this is first line
this is second line
this is third line
Expected is:
Hi this is first line
this is second line
this is third line
What i used is
cat file.txt | sed 's/ //g'
returns,
Hi this is first line
this is second line
this is third line
For a portable sed command use this:
sed 's/^[[:blank:]]*//' file
[[:blank:]] matches space or tab.
EDIT: To remove all spaces using awk:
awk '{$1=$1}1' OFS= file
OR sed:
sed 's/[[:blank:]]*//g' file
sed in your example will replace all the spaces
sed 's/^[ \t]*//' file.txt
cat file.txt | sed -e 's/^[ \t]*//'
OR
sed 's/^[ \t]*//' file.txt
OR if you want to modify file.txt and remove the white spaces in the beginning of the line :
sed -i 's/^[ \t]*//' file.txt
try this line
sed -r 's/^\s*//' file
to remove all spaces(tabs): sed -r 's/\s//g' file
kent$ echo "Hi this is first line
this is second line
this is third line"|sed -r 's/\s//g'
Hithisisfirstline
thisissecondline
thisisthirdline
The most efficient way to remove all blanks from your input is probably not using sed at all, but
tr -d '[:blank:]' < file.txt
That's different from what you have originally asked for, though (removing initial whitespace only).
I am trying to extract text between pattern1 (fixed) and pattern2 (this can be p2-1/p2-2).
can you please tell me how to achieve this in a single command?
A file starts with start and ends with either end or close
File1:
======
junktest
data
start
stackoverflow
sed
close
File2:
======
data2
start
stackoverflow
end
I can extract text from File1 with
sed -n "/start/,/close/p"
And from File2 with
sed -n "/start/,/end/p"
I need a single sed command to achieve both..
something like:
sed -n "/start/, /close or end /p"
Both GNU sed and BSD sed:
sed -nE '/start/,/close|end/p' file
This awk looks better
awk '/start/,/end|close/' file
sed -n -E "/Word1/,/Word2-1/p" | sed -n -E "/Word1/,/Word2-2/p"
Easy with awk:
$ awk '/start/{p=1}p{print}/end|close/{p=0}' file
How i can delete whitespace in each line of file, using bash
For instance, file1.txt. Before:
gg g
gg g
t ttt
after:
gg g
gg g
t ttt
sed -i 's/ //g' your_file will do it, modifying the file inplace.
To delete only the whitespaces at the beginning of one single line, use sed -i 's/^ *//' your_file
In the first expression, we replace all spaces with nothing.
In the second one, we replace at the beginning using the ^ keyword
tr(delete all whitespaces):
$ tr -d ' ' <input.txt >output.txt
$ mv output.txt input.txt
sed(delete leading whitespaces)
$ sed -i 's/^ *//' input.txt
use can use perl -i for in place replacement.
perl -p -e 's/^ *//' file
To delete the white spaces before start of the line if the pattern matches. Use the following command.
For example your foo.in has pattern like this
This is a test
Lolll
blaahhh
This is a testtt
After issuing following command
sed -e '/This/s/ *//' < foo.in > foo.out
The foo.out will be
This is a test
Lolll
blaahhh
This is a testtt
"Whitespace" can include both spaces AND tabs. The solutions presented to date will only match and operate successfully on spaces; they will fail if the whitespace takes the form of a tab.
The below has been tested on the OP's specimen data set with both spaces AND tabs, matching successfully & operating on both:
sed 's/^[[:blank:]]*//g' yourFile
After testing, supply the -i switch to sed to make the changes persistent-