I need to read through some gigantic log files on a Linux system. There's a lot of clutter in the logs. At the moment I'm doing something like this:
cat logfile.txt | grep -v "IgnoreThis\|IgnoreThat" | less
But it's cumbersome -- every time I want to add another filter, I need to quit less and edit the command line. Some of the filters are relatively complicated and may be multi-line.
I'd like some way to apply filters as I am reading through the log, and a way to save these filters somewhere.
Is there a tool that can do this for me? I can't install new software so hopefully it's something that would already be installed -- e.g., less, vi, something in a Python or Perl lib, etc.
Changing the code that generates the log to generate less is not an option.
Use &pattern command within less.
From the man page for less
&pattern
Display only lines which match the pattern; lines which do not
match the pattern are not displayed. If pattern is empty (if
you type & immediately followed by ENTER), any filtering is
turned off, and all lines are displayed. While filtering is in
effect, an ampersand is displayed at the beginning of the
prompt, as a reminder that some lines in the file may be hidden.
Certain characters are special as in the / command:
^N or !
Display only lines which do NOT match the pattern.
^R Don't interpret regular expression metacharacters; that
is, do a simple textual comparison.
Try the multitail tool - as well as letting you view multile logs at once, I'm pretty sure it lets you apply regex filters interactively.
Based on ghostdog74's answer and the less manpage, I came up with this:
~/.bashrc:
export LESSOPEN='|~/less-filter.sh %s'
export LESS=-R # to allow ANSI colors
~/less-filter.sh:
#!/bin/sh
case "$1" in
*logfile*.log*) ~/less-filter.sed < $1
;;
esac
~/less-filter.sed:
/deleteLinesLikeThis/d # to filter out lines
s/this/that/ # to change text on lines (useful to colorize using ANSI escapes)
Then:
less logfileFooBar.log.1 -- applies the filter applies automatically.
cat logfileFooBar.log.1 | less -- to see the log without filtering
This is adequate for now but I would still like to be able to edit the filters on the fly.
see the man page of less. there are some options you can use to search for words for example. It has line editing mode as well.
There's an application by Casstor Software Solutions called LogFilter (www.casstor.com) that can edit Windows/Mac/Linux text files and can easily perform file filtering. It supports multiple filters as well as regular expressions. I think it might be what you're looking for.
Related
I would like to replace parts of one line with the result of the selection being piped into a command.
For example:
echo "hello $(echo "world" | base64)" | vim -
This will open a buffer with hello d29ybGQK in it. Now press wvw to visually select d29ybGQK.
Then I attempted :!base64 -d and I expected the buffer to contain hello world, which did not happen. Indeed, the whole line was piped into the command, and the whole line was replaced.
Is it possible to replace only the visual selection, and have only that selection piped into the command?
I also attempted c<c-r>=system('base64 -d') but that did not send the visual selection to the command's stdin.
Filtering with ! is always line-wise. Your solution with c and the
expression register is an excellent way to solve this. You only forgot
to pass the input to system(), which is its second optional argument.
Since you just changed the text selected, it went into the " register
automatically. All you need to do is to grab it back and pass it to
system with getreg():
c<C-R>=system('base64 -D', getreg('"'))
Note that base64 may echo a newline at the end. If you want to remove
it, either wrap the whole thing in trim(), a new function in Vim 8, or
use [:-2]:
c<C-R>=trim(system('base64 -D', getreg('"')))
c<C-R>=system('base64 -D', getreg('"'))[:-2]
This is a shorthand for [0:-2], meaning grab everything from character
0 to second-last in the resulting string.
Consider creating a visual map if you use it often:
vnoremap <leader>d c<C-R>=system('base64 -D', getreg('"'))[:-2]<CR>
For historical reasons, the Ex commands are inherently line-based; venerable vi also didn't have visual mode yet. That limitation includes filtering through an external command with :range!; it will always filter complete lines.
manual solution
For simple input like in your example, it's probably easiest to temporarily split the line, filter, and then reassemble. For example:
:substitute/ /\r/ | execute '.!base64 -d' | -1join
plugin solution
For a universal solution, you need to use or implement a plugin that grabs the selected text, filters it (probably through system()), and then replaces the selection with the result.
My SubstituteExpression plugin has a {Visual}g= mapping that can filter through Vimscript expressions, Vim functions and commands, and external commands.
express.vim by Tom McDonald offers an almost identical implementation. It also allows on-the-fly creation of operators via :MapExpress and :MapSubpress, something for which I would use my TextTransform plugin, which you need to install as a dependency, anyway. My plugin offers more advanced (cross-mode) repeats, and the :Ex-command expression variant, but has two large dependencies you also need to install.
Apologies if this has been posted already, for I cannot find an answer, even on the vim wiki.
Is there a way I can run multiple commands in vim command-line mode off of a single :g search?
For example,
:%g/foo/ s/bar/\=#a/g | exe "norm /cat\<enter>\"ayiw"
Which (for what I intend it to do) should, on every line matching foo, replace bar with the contents of register a, and then find the next iteration of cat (even if it is many lines ahead), and put the surrounding word into register a.
Instead, this specific syntax completes the subsitution command using the current contents of the initial a register, and then executes the normal mode command on a single line (after the substitution has been completed).
This example is not my specific use-case but shows one instance where this functionality is useful. I realize I could put it all into a single exe, i.e., %g/foo/exe "norm :s/bar/\\=#a/g\<enter>/cat\<enter>\"ayiw", but I would like to do it the first way, as I feel it is more flexible.
I would prefer to do this using vanilla vim, but if a plugin exists for this, that is an okay alternative. Does anybody know if there is syntax to do such a thing?
Okay a "little bit" dirty, but does this work for you?
:let list = split(execute('g/cat/p'), '\n') | g/foo/ s/bar/\=matchstr(remove(list, 0), '\s\d\+\s\zs.*')/g
It first reads all occurences of cat save them in a list.
Then replace the first bar with the first cat... and so on.
The dirty part ist the matchstr command. the g//p also returns a number for the result so the list looks like this:
1 cat
2 cat
3 cat
...
that's why we have to remove a bit from the front. I would love to hear if someone knows a clean solution for that (I am also interested in a clean vimscript solution, does not have to be a oneliner).
You can do this (at least for multiple :s commands applied to a single :g). Example:
" SHORT STORY TITLES to single word of CapitalizedWords within <h3>s
.,$g/^\L\+$/s/[^A-Z0-9 ]\+//ge|s/\u\+/\L&/ge|s/\<\l\+\>/\u&/ge|s/ \+//ge|s/.*/<h3>&<\/h3>/
Using standard Unix tools how can I search in a text file or output for a word with maybe 1-2 letters transposed or missed?
For example my input
function addtion(number, increment)
return number+increment
end
function additoin(number, increment)
return number+increment
end
I would like to search for addition and match addtion and additoin in my input and tell me about it. Because it's code, checking against dictionary is out of the question.
Currently cat file.txt | grep "addition" will simply yield me nothing.
You can play around with the agrep command. It can perform fuzzy, approximate matches.
The following command worked for me:
agrep -2 addition file
You can't do a fuzzy match with standard grep, but if there are specific misspelling you're interested in, you could construct a regular expression that matches those.
For example:
grep add[it]*on
matches the example misspelling you gave. But that's probably not general enough for your purposes.
A better approach is likely going to be to use some sort of static analysis tool specific to the language the code is in. It might not give you the right spelling, but should be able to tell you where the function name and calls to the function use different spellings.
Try the spell command. Note: You might need a dictionary (usually aspell-en in your distro's repositories).
As the answer says, you should definitely try agrep. In addition, there is a newer and much faster alternative ugrep for fuzzy search. Use -Z2 to allow up to 2 errors:
ugrep -Z2 addition file.txt
An insertion, deletion, or substitution is one error. A transposition (as in additoin) counts as two errors, i.e. two substitutions. Use option -i for case-insensitive search and -w to match whole words.
Try this on linux terminal:
grep -rnw "text" ./
When setting autolist in tcsh.rc, and pressing Tab to complete a filename i get a list of available completions in the format of a "regular" ls command.
is there any way to set tcsh in a way that pressing tab will show a list of available completions in the format of "ls -l"?
or if not, then just sort the "regular" list by date?
pressing tab will show a list of available completions in the format of "ls -l"?
No, this is not possible.
Completion in tcsh works by getting a list of words (ie. my trousers are on fire), checking if one of the word matches, and then insert that word in the commandline.
There is no functionality which says "display this to the user, but insert something else in the commandline". So while you could complete commands with ls -l, this would be fairly useless, since you'll get the entire line in your commandline.
just sort the "regular" list by date?
This is also not possible, since tcsh sorts the completions. You can't disable this ...
Sorry :-( I believe that at least zsh is able to do this, but I'm not sure. Maybe bash can do this as well (but again, not sure).
(This information derived from a careful reading of the manpage, and tw.parse.c in the source code.)
I'd like to have a command I can insert into a command pipeline that adds color escapes to its input according to vim's syntax highlighting capabilities.
For example:
cat somefile.js | vim - <???> | less
The resulting text would be that of somefile.js, but colorized according to how the current vim configuration would do it in-editor.
It occurs to me that this must be possible. I agree that the example up there isn't what a sane man might call exactly useful, but that doesn't mean the idea never is.
I think your idea has one basic flaw: that nobody ever thought about allowing such a thing.
Clearly vim is capable of doing syntax highlighting. But I'll bet you an ice cream cone that if you can manage to get vim to stream text through and process it, that you won't like the results.
Consider what happens when you pipe text through more (or less if you prefer). When it goes to the terminal, these programs display one screenful and wait for you to hit the space bar. But if you redirect stdout to some other place than the terminal, these programs notice this and simply copy their input to their output unchanged.
If vim doesn't notice that you are piping text through, it is likely to send cursor-movement commands that you probably don't want in your output. If vim does notice, it is likely to just pass the text, and not syntax-color it. Only if vim does do the syntax-coloring but does not inject cursor-movement stuff will your idea work.
You could try it. Here's an answer that discusses piping stuff through vim:
Execute a command within Vim from the command line
But I say why not pipe your text through a program that was designed and intended to have text piped through it? Pygments can colorize every major programming language and markup format.
http://pygments.org/
The major advantage I see for your idea: you can customize the way vim does syntax coloring, get it the way you want it, and then also use vim to process your text. But it's probably not that hard to customize Pygments, and it might even be satisfactory out of the box, in which case it would definitely be the easiest way to go. And Pygments not only has ANSI sequence output, it also has HTML output, RTF, LaTeX, etc. So if you get Pygments working the way you want it to, it should be able to output whatever output format you need; vim will only have the ANSI sequence one.
There's a Perl module called Text::VimColor that I've heard will do kinda what you're looking for.
http://search.cpan.org/dist/Text-VimColor/
But let me ask this: Why do want it to go through less? Why not use vim as a perfectly good file viewer? view - will read from standard input in read-only mode.
https://gist.github.com/echristopherson/4090959
Via https://superuser.com/a/554531/7198.
Tried on /etc/passwd and it works surprisingly well!
This might be what you're after
cat filename.sh | vim - -c 'syntax on; syn=bash'
This is ugly, but you could alias this:
alias vim.sh="vim -c 'syntax on; syn=bash'"
Then use like this:
cat filename.sh | vim.sh -
Use vimcat !
wget -O /usr/local/bin/vimcat "https://www.vim.org/scripts/download_script.php?src_id=23422"
chmod 755 /usr/local/bin/vimcat
vimcat /etc/passwd
See also: https://www.vim.org/scripts/script.php?script_id=4325