I need to highlight certain keywords like "fail, failed, error, fatal, missing" over my terminal.
I need this with the output of ALL the commands, not any specific command. I assume I need to tweak my bashrc file for this.
To color I can use:
<input coming to terminal>|grep -P --color=auto 'fail|failed|error|fatal|missing|$'
I tried the following command but not helped:
tail -f $(tty) |grep -P --color=auto 'fail|failed|error|fatal|missing|$' &
[1]+ Stopped(SIGTTIN) tail -f $(tty) | grep -P --color=auto 'fail|failed|error|fatal|missing|$'
Searched SO for answers but could not find any question which provides desired an answer.
I don't think there's really an elegant way to do this using the shell. Ideally, you'd get a terminal emulator with this kind a keyword highlighting built in. You can get some of the way by piping the output of bash through a filter that adds ANSI colour escapes. Here is a sed script, that replaces "fail" with (red)fail(normal):
s/fail/\x1B[31m&\x1B[0m/
t done
:done
Run bash with its output piped through sed like this:
$bash | sed -f color.sed
This mechanism is not without problems, but it works in some cases. Usually it's better just to collect up the output you want, and then pipe it through sed, rather than working directly with the bash output.
Related
I need to use the output of a command, which is:
nmap -n -Pn -sS --vv <*IPs*>
The output of this command is devided into 2 parts; first a discovery and after that a port scan, both of them separated by the very first line that says
Nmap scan report for <FirstIP>
What I need is the output of this first part, which by the way is the fastest one, and I need to pipe it to a further command/s (awk, grep or whatever) to filter only the IP addresses. Obviously, the intention of doing this is the need of stopping the running command exactly when the line "Nmap scan report for <*FirstIP*>" appears on the shell (first, because I don't need the other part and second, because the other part takes too much time!)
I found a very close solution here but it didn't worked because it executes both commands (nmap and awk) but there's no output in stdout in shell.
I would be looking for something like this:
nmpa -n -Pn -sS --vv <*IPs*> | awk '/Not shown/ {system("pkill nmap")}' | awk '/^Discovered/{print $6}'
But obviously this doesn't work.
Any ideas?
Most flavors of awk do buffering. There's no option to do what you're doing in gawk, but if you use mawk, you can give it the -Winteractive option, which does not buffer.
Incidentally, you are running two awks, but you only need one:
nmap -n -Pn -sS --vv <*IPs*> | mawk -Winteractive '/Not shown/ {system("pkill nmap")} /^Discovered/{print $6}'
Every predicate in awk runs the associated block. (Although I love awk, this use case might match expect better.)
Given this command line:
left-side | right-side
The problems you have are:
The shell will buffer the output of any command going to a pipe so you may not see "Not shown" in the right-side command until after the left-side command has finished running, and
You want the left-side command to stop running as soon as the right-side command sees "Not shown"
For the first problem, use stdbuf or similar.
For the second - exiting the right-side command will send a terminate signal back to the left-side command so you don't need to do anything else, what you want to happen will simply happen when you exit the right-side command.
So your command line would be something like:
nmap -n -Pn -sS --vv <*IPs*> |
stdbuf awk '/Not shown/{exit} /^Discovered/{print $6}'
idk if you meant to use "Not shown" or "Nmap scan report for" in your sample code. Use whichever is the string you want awk to exit at.
I am quite new to shell scripting.
I am scraping a website and the scraped text contains a lot of repetitions. Usually they are the menus on a forum, for example. Mostly, I do this in Python, but I thought that sed command will save me reading and printing the input, loops etc. I want to delete thousands of repeated lines from the same single file. I do not want to copy it to another file, because I will end up with 100 new files. The following is a shadow script which I run from the bash shell.
#!/bin/sed -f
sed -i '/^how$/d' input_file.txt
sed -i '/^is test$/d' input_file.txt
sed -i '/^repeated text/d' input_file.txt
This is the content of the input file:
how to do this task
why it is not working
this is test
Stackoverflow is a very helpful community of programmers
that is test
this is text
repeated text is common
this is repeated text of the above line
Then I run in the shell the following command:
sed -f scriptFile input_file.txt
I get the following error
sed: scriptFile line 2: untermindated `s' command
How can I correct the script, and what is the correct syntax of the command I should use to get it work?
Any help is highly appreciated.
assuming you know what your script is doing, it's very easy to put them into a script. in your case, the script should be:
/^how$/d
/^is test$/d
/^repeated text/d
that's good enough.
to make the script alone to be executable is easy too:
#!/usr/bin/env sed -f
/^how$/d
/^is test$/d
/^repeated text/d
then
chmod +x your_sed_script
./your_sed_script <old >new
here is a very good and compact tutorial. you can learn a lot from it.
following is an example from the site, just in case the link is dead:
If you have a large number of sed commands, you can put them into a file and use
sed -f sedscript <old >new
where sedscript could look like this:
# sed comment - This script changes lower case vowels to upper case
s/a/A/g
s/e/E/g
s/i/I/g
s/o/O/g
s/u/U/g
Wouldn't it be easier to do it with egrep followed by a mv, for example
egrep -v 'pattern1|pattern2|pattern3|...' <input_file.txt >tmpfile.txt
mv tmpfile.txt input_file.txt
Each pattern would describe the lines being deleted, much like in sed. You would not end up with additional files, because the mv removes them.
If you have so many pattern, that you don't want to specify them directly on the command line, you can store them in a file use the -f option of egrep.
I have this
exec 5<>/dev/tcp/twitter.ca/80
echo -e "GET / HTTP/1.0\n" >&5
cat <&5
I looked a similar script
curl http://cookpad.com 2>&1 | grep -o -E 'href="([^"#]+)"' | cut -d'"' -f2
but I need to use the sed command only.
the output i get is this
sed: -e expression #1, char 2: extra characters after command
#!/bin/bash
exec 5<>/dev/tcp/twitter.ca/80
echo -e "GET / HTTP/1.0\n" >&5
cat <&5 | sed -r -e 'href="([^"#]+)"'
Is what I currently have and I guess what im trying to do is how to use sed to strip it of all extras and keep it with just the htmls?
my output should be look something like this:
href="UnixFortune.apk"
href="UnixFortune-1.0.tgz"
href="BeagleCar.apk"
href="BeagleCar.zip"
sed is a scripting language. Your command looks like you are trying to use the h command (copy pattern to hold space) with options starting with ref=... but the h command doesn't take any options.
Anyway, the command you want is the s command, which performs substitutions. Namely, you want to substitute everything before and after the matching group with nothing (and thus print only the captured group).
sed -r -e 's/.*href="([^"#]+)".*/\1/'
However, this still doesn't do the right thing if there are multiple matches on a line (or lines without a match, although that is easy to fix with sed -n 's/.../p'). You can certainly solve that in sed, but I would suggest you go with grep -o instead, unless you specifically want to learn, write, and maintain sed script. (Or, alternatively, rewrite into an Awk or Perl script. Perl in particular has a lot more leverage for tasks like this.)
And of course, for this particular task, the proper tool is an HTML parser. There is no way to properly pick apart HTML using just regular expressions. See e.g. How to extract links from a webpage using lxml, XPath and Python?
I'm new to bash scripting and I've been learning as I go with a small project I'm taking on. However, I've run into a problem that I cannot seem to get past.
I have a variable that I need to include in a command. When ran directly in the shell (with the variable manually typed), the command returns the expected result. However, I can't get it to work when using a variable.
So, if I manually run this, it correctly returns 0 or 1, depending if it is running or not.
ps -ef | grep -v grep | grep -c ProcessName
However, when I try to embed that into this while clause, it always evaluates to 0 because it's not searching for the correct text.
while [ `ps -ef | grep -v grep | grep -c {$1}` -ne 0 ]
do
sleep 5
done
Is there a way I can accomplish this? I've tried a myriad of different things to no avail. I also tried using the $() syntax for command substitution, but I had no luck with that either.
Thanks!
I think that instead of {$1} you mean "$1". Also, you can just do pgrep -c "$1" instead of the two pipes.
In addition, there's also no need to compare the output of grep -c with 0, since you can just see if the command failed or not. So, a much simplified version might be:
while pgrep "$1" > /dev/null
do
sleep 4
done
You should really use -C with ps rather than the messy pipes if you're using the full process name. If you're interested in substring matching, then your way is the only thing I can think of.
I'd like to know if I can colour highlight the output of a shell command that matches certain strings.
For example, if I run myCommand, with the output below:
> myCommand
DEBUG foo bar
INFO bla bla
ERROR yak yak
I'd like all lines matching ^ERROR\s.* to be highlighted red.
Similarly, I'd like the same highlighting to be applied to the output of grep, less etc...
EDIT: I probably should mention that ideally I'd like to enable this feature globally via a 'profile' option in my .bashrc.
There is an answer in superuser.com:
your-command | grep -E --color 'pattern|$'
or
your-command | grep --color 'pattern\|$'
This will "match your pattern or the end-of-line on each line. Only the pattern is highlighted..."
You can use programs such as:
spc (Supercat)
grc (Generic Colouriser)
highlight
histring
pygmentize
grep --color
You can do something like this, but the commands won't see a tty (some will refuse to run or behave differently or do weird things):
exec > >(histring -fEi error) # Bash
If you want to enable this globally, you'll want a terminal feature, not a process that you pipe output into, because a pipe would be disruptive to some command (two problems are that stdout and stderr would appear out-of-order and buffered, and that some commands just behave differently when outputting to a terminal).
I don't know of any “conventional” terminal with this feature. It's easily done in Emacs, in a term buffer: configure font-lock-keywords for term-mode.
However, you should think carefully whether you really want that feature all the time. What if the command has its own colors (e.g. grep --color, ls --color)? Maybe it would be better to define a short alias to a colorizer command and run myCommand 2>&1|c when you want to colorize myCommand's output. You could also alias some specific always-colorize commands.
Note that the return status of a pipeline is its last command, so if you run myCommand | c, you'll get the status of c, not myCommand. Here's a bash wrapper that avoids this problem, which you can use as w myCommand:
w () {
"$#" | c
return $PIPESTATUS[0]
}
You could try (maybe needs a bit more escaping):
BLUE="$(tput setaf 4)"
BLACK="$(tput sgr0)"
command | sed "s/^ERROR /${BLUE}ERROR ${BLACK}/g"
Try
tail -f yourfile.log | egrep --color 'DEBUG|'
where DEBUG is the text you want to highlight.
You can use the hl command avalaible on github :
git clone http://github.com/mbornet-hl/hl
Then :
myCommand | hl -r '^ERROR.*'
You can use the $HOME/.hl.cfg configuration file to simplify the command line.
hl is written in C (source is available).
You can use up to 42 differents colors of text.
Use awk.
COLORIZE_AWK_COMMAND='{ print $0 }'
if [ -n "$COLORIZE" ]; then
COLORIZE_AWK_COMMAND='
/pattern1/ { printf "\033[1;30m" }
/pattern2/ { printf "\033[1;31m" }
// { print $0 "\033[0m"; }'
fi
then later you can pipe your output
... | awk "$COLORIZE_AWK_COMMAND"
printf is used in the patterns so we don't print a newline, just set the color.
You could probably enable it for specific commands using aliases and user defined shell functions wihtout too much trouble. If your coloring errors I assume you want to process stderr. Since stderr in unbuffered you would probably want to line buffer it by sending through a fifo.