I have gotten comfortable using expressions in Vim to do replaces across large files and would like to know what utility program to learn that will allow me to do such a thing across folders of files in a similar fashion.
What command line program for search and replace is most like vims and works across folders?
sed -i.bak 's/foo/bar/g' *
will replace every foo with bar on every line in every file in the current directory in place and creates a backup file with .bak extension.
So I'd look into sed. But... Vim is very capable:
vim -c "bufdo!%s/foo/bar/g" -c "wqa" *
is almost the same.
bufdo! executes the following command (%s/foo/bar/g in this case => replace every foo with bar on every line (% is a special line address for every line)). wqa means: write all then quit. You can specify at most 10 -c switches to Vim, which will be executed in the specified order. So, basically this is Vim automation:
open every file in the current dir => every file has its own buffer
execute the substitution on every buffer
then write the files and quit
The only difference is the lack of backup file creation (which can be achieved easily with some .vimrc settings). But sed is faster.
probably, you should use vim not external command line program. see :help :find or :help :grep
for example, if you want to grep words across folders, pass **/*.txt.
you could do this by combining perl and bash
for example:
find . -type f -name "*.cpp" -print | xargs perl -i -pe 's/pattern/replace/g'
this will find all '.cpp' files starting at '.' and pass (pipe) each of them (the path) to perl command
Related
I have found some similar questions here but not this specific one and I do not want to break all my files. I have a list of files and I simply need to replace all spaces with underscores. I know this is a sed command but I am not sure how to generically apply this to every file.
I do not want to rename the files, just modify them in place.
Edit: To clarify, just in case it's not clear, I only want to replace whitespace within the files, file names should not be changed.
find . -type f -exec sed -i -e 's/ /_/g' {} \;
find grabs all items in the directory (and subdirectories) that are files, and passes those filenames as arguments to the sed command using the {} \; notation. The sed command it appears you already understand.
if you only want to search the current directory, and ignore subdirectories, you can use
find . -maxdepth 1 -type f -exec sed -i -e 's/ /_/g' {} \;
This is a 2 part problem. Step 1 is providing the proper sed command, 2 is providing the proper command to replace all files in a given directory.
Substitution in sed commands follows the form s/ItemToReplace/ItemToReplaceWith/pattern, where s stands for the substitution and pattern stands for how the operation should take place. According to this super user post, in order to match whitespace characters you must use either \s or [[:space:]] in your sed command. The difference being the later is for POSIX compliance. Lastly you need to specify a global operation which is simply /g at the end. This simply replaces all spaces in a file with underscores.
Substitution in sed commands follows the form s/ItemToReplace/ItemToReplaceWith/pattern, where s stands for the substitution and pattern stands for how the operation should take place. According to this super user post, in order to match whitespace characters you must use either just a space in your sed command, \s, or [[:space:]]. The difference being the last 2 are for whitespace catching (tabs and spaces), with the last needed for POSIX compliance. Lastly you need to specify a global operation which is simply /g at the end.
Therefore, your sed command is
sed s/ /_/g FileNameHere
However this only accomplishes half of your task. You also need to be able to do this for every file within a directory. Unfortunately, wildcards won't save us in the sed command, as * > * would be ambiguous. Your only solution is to iterate through each file and overwrite them individually. For loops by default should come equipped with file iteration syntax, and when used with wildcards expands out to all files in a directory. However sed's used in this manner appear to completely lose output when redirecting to a file. To correct this, you must specify sed with the -i flag so it will edit its files. Whatever item you pass after the -i flag will be used to create a backup of the old files. If no extension is passed (-i '' for instance), no backup will be created.
Therefore the final command should simply be
for i in *;do sed -i '' 's/ /_/g' $i;done
Which looks for all files in your current directory and echos the sed output to all files (Directories do get listed but no action occurs with them).
Well... since I was trying to get something running I found a method that worked for me:
for file in `ls`; do sed -i 's/ /_/g' $file; done
I have a requirement to batch edit a bunch of files using vim based on their content. The simplest example is that I'd like to perform a series of let's say substitutions on files but only if the first line of the file matches a certain pattern.
I'm trying to do this kind of thing:
vim -e -s $file < changes.vim
I should add that I have no access to tools like sed and awk and would like to perform the entire operation in vim.
I recommend that you find the list of files you need, and pass that list into the command you want. For this, a combination of awk and xargs would seem useful. There are probably clever shorter things you can do…
awk 'FNR>1 {nextfile} /pattern/ { print FILENAME ; nextfile }' filePattern | xargs -I{} vim -e -s {} < changes.vim
In the above, filePattern gives all the files you want (maybe *.c), /pattern/ is the regex of the match you are looking for. xargs will take "one output at a time" and substitute it into the following command at the place where I put the {}.
I want to give a tip of the hat to this link where I found the inspiration for this answer.
vim only solution
EDIT - after I posted this you said you need a "vim only" solution. Here it is…
Step 1: create a conditionalEdits.vim file with the following lines at the start:
let line_num = search('searchExpression') " any regex
if line_num == 1 " first line matched
center " put your editing commands here...
update " save changes
endif
quit
Of course, instead of just centering the first line, you will want to put all your editing commands inside the if statement.
Now, you execute this command with
vim -c '/path/to/my/conditionalEdits.vim' -s filePattern
where filePattern matches all the files you might be interested in (but you will know for sure after you have looked at line 1 inside…)
Obviously you can navigate through the file in the usual way and look for matches / patterns etc to your heart's content - but this is the basic idea.
Helpful links: http://www.ibm.com/developerworks/library/l-vim-script-1/
and http://learnvimscriptthehardway.stevelosh.com
I highly recommend that you do this in a separate directory, using copies of a handful of files first, to make sure this actually does what you think it does. I would hate to be responsible for a bunch of files being overwritten (you do back up, right?)
You can loop over all files, if you find the pattern, open vim. Once it is modified to your needs and closed, the next one will open.
#!/usr/bin/env bash
for file in *; do
if [[ "$(sed '1q' ${file})" == "pattern" ]]; then
vim ${file}
fi
done
Within Vim, you can determine the matching files via :vimgrep; to check for a match in the first line, the \%l atom is handy:
:vimgrep /\%1lcertain pattern/ {file-glob}
Then, you can iterate through all matches with :cfnext, or use the :QFDo command from here.
You can pass those commands either via vim -c {cmd} -c {cmd} ..., or in a separate script, as you outline in your question.
I am trying to remove 2 lines from all my Javascript files on my Linux shared hosting. I wanted to do this without writing a script as I know this should be possible with sed. My current attempt looks like this:
find . -name "*.js" | xargs sed -i ";var
O0l='=sTKpUG"
The second line is actually longer than this but is malicious code so I have not included it here. As you guessed my server has been hacked so I need to clean up all these JavaScript files.
I forgot to mention that the output I am getting at the moment is:
sed: -e expression #1, char 4: expected newer version of sed
The 2 lines are just as follows consecutively:
;var
O0l='=sTKpUG
except that the second line is longer, but the rest of the second line should not influence the command.
He meant removing two adjacent lines.
you can do something like this, remember to backup your files.
find . -name "*.js" | xargs sed -i -e "/^;var/N;/^;var\nO0l='=sTKpUG/d"
Since sed processes input file line by line, it does not store the newline '\n' character in its buffer, so we need to tell it by using flag /N to append the next line, with newline character.
/^;var/N;
Then we do our pattern searching and deleting.
/^;var\nO0l='=sTKpUG/d
It really isn't clear yet what the two lines look like, and it isn't clear if they are adjacent to each other in the JavaScript, so we'll assume not. However, the answer is likely to be:
find . -name "*.js" |
xargs sed -i -e '/^distinctive-pattern1$/d' -e '/^alternative-pattern-2a$/d'
There are other ways of writing the sed script using a single command string; I prefer to use separate arguments for separate operations (it makes the script clearer).
Clearly, if you need to keep some of the information on one of the lines, you can use a search pattern adjusted as appropriate, and then do a substitute s/short-pattern// instead of d to remove the short section that must be removed. Similarly with the long line if that's relevant.
Is there a way to execute a Vim command on a file from the command line?
I know the opposite is true like this:
:!python %
But what if I wanted to :retab a file without opening it in Vim? For example:
> vim myfile.c
:retab | wq
This will open myfile.c, replace the tabs with spaces, and then save and close. I'd like to chain this sequence together to a single command somehow.
It would be something like this:
> vim myfile.c retab | wq
This works:
gvim -c "set et|retab|wq" foo.txt
set et (= set expandtab) ensures the tab characters get replaced with the correct number of spaces (otherwise, retab won't work).
I don't normally use it, but vim -c ... also works
The solution as given above presumes the default tab stop of eight is appropriate. If, say, a tab stop of four is intended, use the command sequence "set ts=4|set et|retab|wq".
You have several options:
-c "commands" : will play Ex commands as you entered them in the command line.
In your example : vim myfile -c 'retab | wq'. This is what Firstrock suggested.
-S "vim source file" : will source given vim script
(like running vim -c "source 'vim source file'"):
If you have a file script.vim containing:
retab
wq
Then you can use vim myfile.c -s script.vim (the extension does not really matter)
-s "scriptin file": will play contents of file as it contains normal mode commands: If you have script.txt containing:
:retab
ZZ
with end of lines consisting of a single ^M character (for example you saved the script using the :set fileformat=mac | w), then you can run: vim myfile.c -S script.txt (ZZ is another way to exit vim and save current file).
Note that you can record those scripts with vim my_file -W script.txt, but it suffers a bug if you happen to use gvim (the GUI).
Not a direct answer to your question, but if you want to replace tabs with spaces (or do any other regex search/replace) for a list of files, you can just use in-place sed search/replace:
sed -i 's/\t/ /g' foo1.txt foo2.txt
or
ls *.txt | xargs sed -i 's/\t/ /g'
(In this example I am replacing each tab character with three spaces.)
NOTE: the -i flag means operate in-place.
From the sed man page:
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension
supplied)
I want to get automatically to the positions of the results in Vim after grepping, on command line. Is there such feature?
Files to open in Vim on the lines given by grep:
% grep --colour -n checkWordInFile *
SearchToUser.java:170: public boolean checkWordInFile(String word, File file) {
SearchToUser.java~:17: public boolean checkWordInFile(String word, File file) {
SearchToUser.java~:41: if(checkWordInFile(word, f))
If you pipe the output from grep into vim
% grep -n checkWordInFile * | vim -
you can put the cursor on the filename and hit gF to jump to the line in that file that's referenced by that line of grep output. ^WF will open it in a new window.
From within vim you can do the same thing with
:tabedit
:r !grep -n checkWordInFile *
which is equivalent to but less convenient than
:lgrep checkWordInFile *
:lopen
which brings up the superfantastic quickfix window so you can conveniently browse through search results.
You can alternatively get slower but in-some-ways-more-flexible results by using vim's native grep:
:lvimgrep checkWordInFile *
:lopen
This one uses vim REs and paths (eg allowing **). It can take 2-4 times longer to run (maybe more), but you get to use fancy \(\)\#<=s and birds of a feather.
Have a look at "Grep search tools integration with Vim" and "Find in files within Vim". Basically vim provides these commands for searching files:
:grep
:lgrep
:vimgrep
:lvimgrep
The articles feature more information regarding their usage.
You could do this:
% vim "+/checkWordInFile" $(grep -l checkWordInFile *)
This will put in the vim command line a list of all the files that match the regex. The "+/..." option will tell vim to search from the start of each file until it finds the first line that matches the regex.
Correction:
The +/... option will only search the first file for the regex. To search in every file you need this:
% vim "-c bufdo /checkWordInFile" $(grep -l checkWordInFile *)
If this is something you need to do often you could write a bash function so that you only need to specify the regex once (assuming that the regex is valid for both grep and vim).
I think this is what you are looking for:
http://www.vim.org/scripts/script.php?script_id=2184
When you open a file:line, for instance when coping and pasting from an error from your compiler (or grep output) vim tries to open a file with a colon in its name. With this little script in your plugins folder if the stuff after the colon is a number and a file exists with the name especified before the colon vim will open this file and take you to the line you wished in the first place.
It's definitely what I was looking for.
I highly recommend ack.vim over grep for this functionality.
http://github.com/mileszs/ack.vim
http://betterthangrep.com/
You probably want to make functions for these. :)
Sequential vim calls (console)
grep -rn "implements" app | # Or any (with "-n") you like
awk '{
split($0,a,":"); # split on ":"
print "</dev/tty vim", a[1], "+" a[2] # results in lines with "</dev/tty vim <foundfile> +<linenumber>
}' |
parallel --halt-on-error 1 -j1 --tty bash -ec # halt on error and "-e" important to make it possible to quit in the middle
Use :cq from vim to stop editing.
Concurrent opening in tabs (gvim)
Start the server:
gvim --servername GVIM
Open the tabs:
grep -rn "implements" app | # again, any grep you like (with "-n")
awk "{ # double quotes because of $PWD
split(\$0,a,\":\"); # split on ":"
print \":tabedit $PWD/\" a[1] \"<CR>\" a[2] \"G\" # Vim commands. Open file, then jump to line
}" |
parallel gvim --servername GVIM --remote-send # of course the servername needs to match
If you use git, results are often more meaningful when you search only in the files tracked by git. To open files at the given line which is a search result of git grep in vim you will need the fugitive plugin, then
:copen
:Ggrep pattern
Will give you the list in a buffer and you can choose to open files from your git grep results.
In this particular example:
vim SearchToUser.java +170