I have this file
foo
foo bar
foo bar baz
bar baz
foo baz
baz bar
bar
baz
foo 42
foo bar 42 baz
baz 42
I want to
Select lines which contain foo and do NOT contain bar
Delete lines which contain foo and do NOT contain bar
I read somewhere (can't find the link) that I have to use :exec with | for this.
I tried the following, but it doesn't work
:exec "g/foo" # works
:exec "g/foo" | exec "g/bar" -- first returns lines with foo, then with bar
:exec "g/foo" | :g/bar -- same as above
And ofcourse if I cannot select a line, I cannot execute normal dd on it.
Any ideas?
Edit
Note for the bounty:
I'm looking for a solution that uses proper :g and :v commands, and does not use regex hacks, as the conditions may not be the same (I can have 2 includes, 3 excludes).
Also note that the last 2 examples of things that don't work, they do work for just deleting the lines, but they return incorrect information when I run them without deleting (ie, viewing the selected lines) and they behave as mentioned above.
I'm no vim wizard, but if all you want to do is "Delete lines which contain foo and do NOT contain bar" then this should do (I tried on your example file):
:v /bar/s/.*foo.*//
EDIT: actually this leaves empty lines behind. You probably want to add an optional newline to that second search pattern.
This might still be hackish to you, but you can write some vimscript to make a function and specialized command for this. For example:
command! -nargs=* -range=% G <line1>,<line2>call MultiG(<f-args>)
fun! MultiG(...) range
let pattern = ""
let command = ""
for i in a:000
if i[0] == "-"
let pattern .= "\\(.*\\<".strpart(i,1)."\\>\\)\\#!"
elseif i[0] == "+"
let pattern .= "\\(.*\\<".strpart(i,1)."\\>\\)\\#="
else
let command = i
endif
endfor
exe a:firstline.",".a:lastline."g/".pattern."/".command
endfun
This creates a command that allows you to automate the "regex hack". This way you could do
:G +foo -bar
to get all lines with foo and not bar. If an argument doesn't start with + or - then it is considered the command to add on to the end of the :g command. So you could also do
:G d +foo -bar
to delete the lines, or even
:G norm\ foXp +two\ foos -bar
if you escape your spaces. It also takes a range like :1,3G +etc, and you can use regex in the search terms but you must escape your spaces. Hope this helps.
This is where regular expressions get a bit cumbersome. You need to use the zero width match \(search_string\)\#=. If you want to match a list of items in any order, the search_string should start with .* (so the match starts from the start of the line each time). To match a non-occurrence, use \#! instead.
I think these commands should do what you want (for clarity I am using # as the delimiter, rather than the usual /):
Select lines which contain foo and bar:
:g#\(.*foo\)\#=\(.*bar\)\#=
Select lines which contain foo, bar and baz
:g#\(.*foo\)\#=\(.*bar\)\#=\(.*baz\)\#=
Select lines which contain foo and do NOT contain bar
:g#\(.*foo\)\#=\(.*bar\)\#!
Delete lines which contain foo and bar
:g#\(.*foo\)\#=\(.*bar\)\#=#d
Delete lines which contain foo and do NOT contain bar
:g#\(.*foo\)\#=\(.*bar\)\#!#d
You won't achieve your requirements unless you're willing to use some regular expressions since the expressions are what drives :global and it's opposite :vglobal.
This is no hacking around but how the commands are supposed to work: they need an expression to work with. If you're not willing to use regular expressions, I'm afraid you won't be able to achieve it.
Answer terminates here if you're not willing to use any regular expressions.
Assuming that we are nice guys with an open mind, we need a regular expression that is true when a line contains foo and not bar.
Suggestion number 5 of Prince Goulash is quite there but doesn't work if foo occurs after bar.
This expression does the job (i.e. print all the lines):
:g/^\(.*\<bar\>\)\#!\(.*\<foo\>\)\#=/
If you want to delete them, add the delete command:
:g/^\(.*\<bar\>\)\#!\(.*\<foo\>\)\#=/d
Description:
^ starting from the beginning of the line
\(.*\<bar\>\) the word bar
\#! must never appear
\(.*\<foo\>\)\#= but the word foo has to appear anywhere on the line
The two patterns could also be swapped:
:g/^\(.*\<foo\>\)\#=\(.*\<bar\>\)\#!/
yields the same results.
Tested with the following input:
01 foo
02 foo bar
03 foo bar baz
04 bar baz
05 foo baz
06 baz bar
07 bar
08 baz
09 foo 42
10 foo bar 42 baz
11 42 foo baz
12 42 foo bar
13 42 bar foo
14 baz 42
15 baz foo
16 bar foo
Regarding multiple includes/excludes:
Each exclude is made of the pattern
\(.*\<what_to_exclude\>\)\#!
Each include is made of the pattern
\(.*\<what_to_include\>\)\#=
To print all the lines that contain foo but not bar nor baz:
g/^\(.*\<bar\>\)\#!\(.*\<baz\>\)\#!\(.*\<foo\>\)\#=/
Print all lines that contain foo and 42 but neither bar nor baz:
g/^\(.*\<bar\>\)\#!\(.*\<baz\>\)\#!\(.*\<foo\>\)\#=\(.*\<42\>\)\#=/
The sequence of the includes and excludes is not important, you could even mix them:
g/^\(.*\<bar\>\)\#!\(.*\<42\>\)\#=\(.*\<baz\>\)\#!\(.*\<foo\>\)\#=/
One might think a combination like :g/foo/v/bar/d would work, but unfortunately this isn't possible, and you will have to recur to one of the proposed work-arounds.
As described in the help, behind the scenes the :global command works in two stages,
first marking the lines on which to operate,
then performing the operation on them.
Out of interest, I had a look at the relevant parts in the Vim source: In ex_cmds.c, ex_global(), you will find that the global flag global_busy prevents repeated execution of the command while it is busy.
You want to employ a negative look ahead. This article gives more or less the specific example you are trying to achieve.
http://www.littletechtips.com/2009/09/vim-negative-match-using-negative-look.html
I changed it to
:g/foo(.*bar)\#!/d
Please let us know if you consider this a regex hack.
I will throw my hat in the ring. As vim's documentation explicitly states recursive global commands are invalid and the regex solution will get pretty hairy quickly, I think this is job for a custom function and command. I have created the :G command.
The usage is as :G followed by patterns surrounded by /. Any pattern that should not match is prefixed with a !.
:G /foo/ !/bar/ d
This will delete all lines that match /foo/ and does not match /bar/
:G /42 baz/ !/bar/ norm A$
This will append a $ to all lines matching /42 baz/ and that don't match /bar/
:G /foo/ !/bar/ !/baz/ d
This will delete all lines that match /foo/ and does not match /bar/ and does not match /baz/
The script for the :G command is below:
function! s:ManyGlobal(args) range
let lnums = {}
let patterns = []
let cmd = ''
let threshold = 0
let regex = '\m^\s*\(!\|v\)\=/.\{-}\%(\\\)\#<!/\s\+'
let args = a:args
while args =~ regex
let pat = matchstr(args, regex)
let pat = substitute(pat, '\m^\s*\ze/', '', '')
call add(patterns, pat)
let args = substitute(args, regex, '', '')
endwhile
if args =~ '\s*'
let cmd = 'nu'
else
let cmd = args
endif
for p in patterns
if p =~ '^(!\|v)'
let op = '-'
else
let op = '+'
let threshold += 1
endif
let marker = "let l:lnums[line('.')] = get(l:lnums, line('.'), 0)" . op . "1"
exe a:firstline . "," . a:lastline . "g" . substitute(p, '^(!\|v)', '', '') . marker
endfor
let patterns = []
for k in keys(lnums)
if threshold == lnums[k]
call add(patterns, '\%' . k . 'l')
endif
endfor
exe a:firstline . "," . a:lastline . "g/\m" . join(patterns, '\|') . "/ " . cmd
endfunction
command! -nargs=+ -range=% G <line1>,<line2>call <SID>ManyGlobal(<q-args>)
The function basically parses out the arguments then goes and marks all matching lines with each given pattern separately. Then executes the given command on each line that is marked the proper amount of times.
All right, here's one which actually simulates recursive use of global commands. It allows you to combine any number of :g commands, at least theoretically. But I warn you, it isn't pretty!
Solution to the original problem
I use the Unix program nl (bear with me!) to insert line numbers, but you can also use pure Vim for this.
:%!nl -b a
:exec 'norm! qaq'|exec '.,$g/foo/d A'|exec 'norm! G"apddqaq'|exec '.,$v/bar/d'|%sort|%s/\v^\s*\d+\s*
Done! Let's see the explanation and general solution.
General solution
This is the approach I have chosen:
Introduce explicit line numbering
Use the end of the file as a scratch space and operate on it repeatedly
Sort the file, remove the line numbering
Using the end of the file as a scratch space (:g/foo/m$ and similar) is a pretty well-known trick (you can find it mentioned in the famous answer number one). Also note that :g preserves relative ordering of the lines – this is crucial. Here we go:
Preparation: Number lines, clear "accumulator" register a.
:%!nl
qaq
The iterative bit:
:execute global command, collect matching lines by appending them into the accumulator register with :d A.
paste the collected lines at the end of the file
repeat for range .,$ (the scratch space, or in our case, the "match" space)
Here's an extended example: delete lines which do contain 'foo', do not contain 'bar', do contain '42' (just for the demonstration).
:exec '.,$g/foo/d A' | exec 'norm! G"apddqaq' | exec '.,$v/bar/d A' | exec 'norm! G"apddqaq' | exec '.,$g/42/d A' | exec 'norm! G"apddqaq'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(this is the repeating bit)
When the iterative bit ends, the lines .,$ contain the matches for your convenience. You can delete them (dVG) or whatever.
Cleanup: Sort, remove line numbers.
:%sort
:%s/\v^\s*\d+\s*
I'm sure other people can improve on the details of the solution, but if you absolutely need to combine multiple :gs and :vs into one, this seems to be the most promising solution.
The in-built solutions looks very complex.
One easy way would be to use LogiPat plugin:
Doc: http://www.drchip.org/astronaut/vim/doc/LogiPat.txt.html
Plugin: http://www.drchip.org/astronaut/vim/index.html#LOGIPAT
With this, you can easily search for patterns.
For e.g, to search for lines containing foo, and not bar, use:
:LogiPat "foo"&!("bar")
This would highlight all the lines matching the logical pattern (if you have set hls).
That way you can cross-check whether you got the correct lines, and then traverse with 'n', and delete with 'dd', if you wish.
I realize you explicitly stated that you want solutions using :g and :v, but I firmly believe this is a perfect example of a case where you really should use an external tool.
:%!awk '\!/foo/ || /bar/'
There's no need to re-invent the wheel.
Select lines which contain foo and do NOT contain bar
Delete lines which contain foo and do NOT contain bar
This can be done by combining global and substitute commands:
:v/bar/s/.*foo.*//g
Related
How to make Vim write lines into a file according to a very simple arithmetic pattern?
Example:
foo1
foo2
foo3
...
foo99
foo100
I came up with the following solution using Ex commands
:for i in range(1,100)
: execute "normal ofoo" . i
:endfor
but I am convinced there must be something more straightforward.
I would do it with macro.
First type one line:
foo1
then
qqYp<c-a>q
finally replay the macro:
98#q
Something like:
:let l = map(range(1,100), '"foo".v:val')
:put=l
" Unfortunately put won't accept the expression, append() would though
call append(line('.'), map(range(1,100), '"foo".v:val'))
let i=1 | g/aaa\zs/s//\=i/ | let i=i+1
The above command add counter number after matched pattern. So the following text is changed.
aaab
aaab
aaab
to
aaa1b
aaa2b
aaa3b
'|' joints commands into one command. In my opinion, the commands are executed sequentially like firstly let i=1, then g/aaa\zs/s//\=i/ , finally let i=i+1 . From the result above, s//\=i/**and **let i=i+1 are executed by g command. Can anyone explain?
The following command does wrong work. But I don't know why.
let i=1 | g/aaa\zs/s//\=i | let i=i+1
In s//\=i/, the replacement string is terminated and the | is treated as an argument by global. However, when you remove the trailing /, the replacement string to s consumes the | let i=i+1. From the help doc for sub-replace-special, you can find: "When the substitute string starts with "\=" the remainder is interpreted as an expression." So the expression i | let i=i+1 is evaluated, but the increment is not available outside of that evaluation.
You should understand your first command as:
let i=1 | g/aaa\zs/ ( s//\=i/ | let i=i+1 )
(Parenthesis are only here for explaining, they'd cause syntax error if typed).
i.e. everything after the g/<pattern/ is a single command given as an argument to the global g command.
So indeed: we start with let i=1, then for all lines matching pattern aaa we execute: s//\=i/ | let i=i+1 (substitution, then incrementing i).
Your second command does not work because s does not function the same way as g, and it does need an ending / after the expression to substitute to pattern.
Usually, the | separates two Ex commands, and they are then indeed executed sequentially. But some commands take a | as part of their arguments. :global is one of them (full list at :help :bar). So, the special application of commands over matching lines is applied with both the :s and the :let commands (the latter of which can be shortened as :let i+=1 BTW).
I have a file that contains lines as follows:
one one
one one
two two two
one one
three three
one one
three three
four
I want to remove all occurrences of the duplicate lines from the file and leave only the non-duplicate lines. So, in the example above, the result should be:
two two two
four
I saw this answer to a similar looking question. I tried to modify the ex one-liner as given below:
:syn clear Repeat | g/^\(.*\)\n\ze\%(.*\n\)*\1$/exe 'syn match Repeat "^' . escape(getline ('.'), '".\^$*[]') . '$"' | d
But it does not remove all occurrences of the duplicate lines, it removes only some occurrences.
How can I do this in vim? or specifically How can I do this with ex in vim?
To clarify, I am not looking for sort u.
If you have access to UNIX-style commands, you could do:
:%!sort | uniq -u
The -u option to the uniq command performs the task you require. From the uniq command's help text:
-u, --unique
only print unique lines
I should note however that this answer assumes that you don't mind that the output doesn't match any sort order that your input file might have already.
if you are on linux box with awk available, this line works for your needs:
:%!awk '{a[$0]++}END{for(x in a)if(a[x]==1)print x}'
Assuming you are on an UNIX derivative, the command below should do what you want:
:sort | %!uniq -u
uniq only works on sorted lines so we must sort them first with Vim's buit-in :sort command to save some typing (it works on the whole buffer by default so we don't need to pass it a range and it's a built-in command so we don't need the !).
Then we filter the whole buffer through uniq -u.
My PatternsOnText plugin version 1.30 now has a
:DeleteAllDuplicateLinesIgnoring
command. Without any arguments, it'll work as outlined in your question.
It does not preserve the order of the remaining lines, but this seems to work:
:sort|%s/^\(.*\)\n\%(\1\n\)\+//
(This version is #Peter Rincker's idea, with a little correction from me.) On vim 7.3, the following even shorter version works:
:sort | %s/^\(.*\n\)\1\+//
Unfortunately, due to differences between the regular-expression engines, this no longer works in vim 7.4 (including patches 1-52).
Taking the code from here and modifying it to delete the lines instead of highlighting them, you'll get this:
function! DeleteDuplicateLines() range
let lineCounts = {}
let lineNum = a:firstline
while lineNum <= a:lastline
let lineText = getline(lineNum)
if lineText != ""
if has_key(lineCounts, lineText)
execute lineNum . 'delete _'
if lineCounts[lineText] > 0
execute lineCounts[lineText] . 'delete _'
let lineCounts[lineText] = 0
let lineNum -= 1
endif
else
let lineCounts[lineText] = lineNum
let lineNum += 1
endif
else
let lineNum += 1
endif
endwhile
endfunction
command! -range=% DeleteDuplicateLines <line1>,<line2>call DeleteDuplicateLines()
This is not any simpler than #Ingo Karkat's answer, but it is a little more flexible. Like that answer, this leaves the remaining lines in the original order.
function! RepeatedLines(...)
let first = a:0 ? a:1 : 1
let last = (a:0 > 1) ? a:2 : line('$')
let lines = []
for line in range(first, last - 1)
if index(lines, line) != -1
continue
endif
let newlines = []
let text = escape(getline(line), '\')
execute 'silent' (line + 1) ',' last
\ 'g/\V' . text . '/call add(newlines, line("."))'
if !empty(newlines)
call add(lines, line)
call extend(lines, newlines)
endif
endfor
return sort(lines)
endfun
:for x in reverse(RepeatedLines()) | execute x 'd' | endfor
A few notes:
My function accepts arguments instead of handling a range. It defaults to the entire buffer.
This illustrates some of the functions for manipulating lists. :help list-functions
I use /\V (very no magic) so the only character I need to escape in a search pattern is the backslash itself. :help /\V
Add line number so that you can restore the order before sort
:%s/^/=printf("%d ", line("."))/g
sort
:sort /^\d+/
Remove duplicate lines
:%s/^(\d+ )(.*)\n(\d+ \2\n)+//g
Restore order
:sort
Remove line number added in #1
:%s/^\d+ //g
please use perl ,perl can do it easily !
use strict;use warnings;use diagnostics;
#read input file
open(File1,'<input.txt') or die "can not open file:$!\n";my #data1=<File1>;close(File1);
#save row and count number of row in hash
my %rownum;
foreach my $line1 (#data1)
{
if (exists($rownum{$line1}))
{
$rownum{$line1}++;
}
else
{
$rownum{$line1}=1;
}
}
#if number of row in hash =1 print it
open(File2,'>output.txt') or die "can not open file:$!\n";
foreach my $line1 (#data1)
{
if($rownum{$line1}==1)
{
print File2 $line1;
}
}
close(File2);
Does Vimscript allow multi-line strings?
The python and ruby commands allow the format: :python << EOF
Can you do anything similar with strings?
Vimscript does allow continuation of the previous line by starting the next with a backslash, but that isn't quite as convenient as a heredoc string such as you would find in Ruby, PHP, or Bash.
let g:myLongString='A string
\ that has a lot of lines
\ each beginning with a
\ backslash to continue the previous one
\ and whitespace before the backslash
\ is ignored'
Have a look at the relevant documentation on line-continuation.
Vimscript multi-line strings, dot operator:
Enumerating assignments and including the previous lets you concat over lines
let foo = "bar"
let foo = foo . 123
echom foo "prints: bar123
Use a compound string concatenation operator dot equals:
let foo = "bar"
let foo .= 123
echom foo "prints: bar123
Make a list of your strings and numbers and join them:
let foo = ["I'm", 'bat', 'man', 11 ][0:4]
echo join(foo) "prints: I'm bat man 11
Same as above but join with array slicing
let foo = ["I'm", 'bat', 'man', [ "i'm", "a", "mario" ] ]
echo join(foo[0:2]) . " " . join(foo[3])
"prints: I'm bat man i'm a mario
A backslash at the beginning of line allows for line continuation
let foo = "I don't think mazer
\ intends for us to find
\ a diplomatic solution"
echom foo
let foo = 'Keep falling,
\ let the "grav hammer" and environment
\ do the work'
echom foo
Prints:
I don't think mazer intends for us to find a diplomatic solution
Keep falling, let the "grav hammer" and environment do the work
Hide your secret text and most ancient tomes inside a function thustly:
function! myblockcomment()
(*& So we back in the club
// Get that bodies rocking
!## from side to side, side side to side.
!##$ = %^&&*()+
endfunction
The memory location of the free form text is the file where it sits on disk. The function is never run, or the interpreter would puke, so there it until you use vim reflection to get the implementation of myblockcomment() then do whatever you want. Don't do this except to bedazzle and confuse.
You can not use << to create string, but you can use << to create list of string.
see :help :let=<<
below is example from vim doc
let text =<< trim END
if ok
echo 'done'
endif
END
By outside, I want solutions that does not use Vim's scripting hacks but try to reuse certain basic *ix tools. Inside Vim stuff asks for solutions to get the column-increment with inside stuff such as scripting.
1 1
1 2
1 3
1 ---> 4
1 5
1 6
. .
. .
Vim has a script that does column-vise incrementing, VisIncr. It has gathered about 50/50 ups and down, perhaps tasting a bit reinventing-the-wheel. How do you column-increment stuff in Vim without using such script? Then the other question is, how do you column-increment stuff without/outside Vim?
Most elegant, reusable and preferably-small wins the race!
I don't see a need for a script, a simple macro would do
"a yyp^Ayy
then play it, or map to play it.
Of course, there is always the possibility that I misunderstood the question entirely...
The optimal choice of a technique highly depends on the actual circumstances
of the transformation. There are at least two points variations affecting
implementation:
Whether the lines to operate on are the only ones in a file? If not,
is the range of lines defined by context (i.e. it separated by blank
lines, like a paragraph) or is it arbitrary and should be specified by
user?
Are those lines already contain numbers that should be changed or is
it necessary to insert new ones leaving the text on the lines in tact?
Since there is no information to answer these questions, below we will try to
construct a flexible solution.
A general solution is a substitution operating on the beginnings of the lines
in the range specified by the user. Visual mode is probably the simplest way
of selecting an arbitrary range of lines, so we assume here that boundaries of
the range are defined by the visual selection.
:'<,'>s/^\d\+/\=line(".")-line("''")+1/
If it is necessary to number every line in a buffer, the command can be
simplified as follows.
:%s/^\d\+/\=line('.')/
In any case, if the number should be merely inserted at the beginnings of the
lines (without modifying the ones that already exist), one can change the
pattern from ^\d\+ to ^, and optionally add a separator:
:'<,'>s/^\d\+/\=(line(".")-line("''")+1).' '/
or
:%s/^/\=line('.').' '/
respectively.
For a solution based on command-line tools, one can consider using stream
editors like Sed or text extraction and reporting tools like AWK.
To number each of the lines in a file using Sed, run the commands
$ sed = filename | sed 'N;s/\n/ /'
In order to do the same in AWK, use the command
$ awk '{print NR " " $0}' filename
which could be easily modfied to limit numbering to a particular range of lines
satisfying a certain condition. For example, the following command numbers the
lines two through eight.
$ awk '{print (2<=NR && NR<=8 ? ++n " " : "") $0}' filename
Having an interest in how commands similar to those from the script linked in
the question statement are implemented, one can use the following command as
a reference.
vnoremap <leader>i :call EnumVisualBlock()<cr>
function! EnumVisualBlock() range
if visualmode() != "\<c-v>"
return
endif
let [l, r] = [virtcol("'<"), virtcol("'>")]
let [l, r] = [min([l, r]), max([l, r])]
let start = matchstr(getline("'<"), '^\d\+', col("'<")-1)
let off = start - line("'<")
let w = max(map([start, line("'>") + off], 'len("".v:val)'))
exe "'<,'>" 's/\%'.l.'v.*\%<'.(r+1).'v./'.
\ '\=printf("%'.w.'d",line(".")+off).repeat(" ",r-l+1-w)'
endfunction
If you want change 1 1 1 1 ... to 1 2 3 4 .... (Those numbers should be on different lines.)
:let i=1 | g/1/s//\=i/g | let i+=1
If some of 1 1 1 1 ... are in the same line:
:let g:i = 0
:func! Inc()
: let g:i+=1
: return g:i
:endfun
:%s/1/\=Inc()/g