I have a for loop that I pass through ssh to a server that is formatted to look kind of like this...
i=0
for cmd in cmd_list;
do ${cmd} | sed "s/^/OUTPUT_${i}_: /" &
(( i++ ));
done;
wait
The idea is that the for loop will run a list of commands that I give it and pipe each one to sed where sed prepends each line of output with a command number (0, 1, 2, 3, etc...). Then it backgrounds so as to allow parallel execution. This allows me to keep track of which command the output was associated with since the data could come back simultaneously and all mixed up. This works really well. Depending on the timing of when the commands print out information and when they are complete, the output might look something like this...
OUTPUT_0_: some_data_string_from_a_command
OUTPUT_2_: some_data_string_from_a_command
OUTPUT_0_: some_data_string_from_a_command
OUTPUT_3_: some_data_string_from_a_command
OUTPUT_1_: some_data_string_from_a_command
OUTPUT_1_: some_data_string_from_a_command
However, what I really want to do is this...
do ${cmd} 2>&1 | sed "s/^/OUTPUT_${i}_${PIPESTATUS[0]}: /" &
So I can get back this...
OUTPUT_0_0: some_data_string_from_a_command
OUTPUT_2_1: some_error_message_from_a_command
OUTPUT_0_0: some_data_string_from_a_command
OUTPUT_3_1: some_error_message_from_a_command
OUTPUT_1_0: some_data_string_from_a_command
OUTPUT_1_0: some_data_string_from_a_command
This works fine on the first command if it errors out. I will usually get the non-zero exit code from ${PIPESTATUS[0]}. However, when I have purposefully planted commands further along in the list that I know will fail (i.e. cat /tmp/some_non_existent_file), PIPESTATUS does not give me the proper exit code of the command in the pipe chain. I will sometimes get 0 instead of whatever the real exit code is.
Any idea why this is?
Commands in a pipeline are executed in parallel. This means that they may not have exited by the time you evaluate PIPESTATUS, particularly since PIPESTATUS is expanded by the shell before the sed command is actually run.
I don't really understand what it is you are trying to do, because you appear to want to violate causality. Have you thought about it carefully? You seem to want the exit value of a command with its output, but if it is outputting data it clearly hasn't exited. You are relying on a small timing window always being open. It may be for some short-run-time commands, but that is even questionable since coordinating parallel execution without synchronisation operations is prone to "random" failure.
If you want to capture the data along with the output status of a command, you'll need to save the output away somewhere (in a file for instance), and when the process exits, then output the data with the exit status.
Related
When using fish shell in a terminal-emulator (such as terminator) together with a command that outputs lots of text it could be useful to get some color coding on the output. I know that a script can add color code information to the output like "grep --color=auto". I guess it's possible to modify the fish terminal to scan through the output and add this in special places?
What I want to do is that the text "error" appearing in the output from any script always is marked red and that "warning" always is marked yellow. Anyone knows if this is possible by introducing function files in the ~/.config/fish/functions dir or similar?
This is basically a layering violation. Usually the output of external commands does not go back through the shell. It goes directly to the terminal.
Also, anything you do here has the potential to slow output down. (And because of fish issue #1396, this can be rather extreme).
That said, it's possible if you always pipe to a function like this
function colorstuff
while read -l input
switch $input
case "*error*"
set_color red
echo $input
case "*warning*"
set_color yellow
echo $input
case "*"
set_color normal
echo $input
end
end
set_color normal
end
Use it like somecommand | colorstuff. (And maybe add 2>&1 if you also wish to have stderr colored)
In my tests, this causes a noticeable slowdown, and even with that issue fixed it will still be slower since it has to match every single line.
Really, the real solution is for whatever tool you are using to color itself, since it knows what the output means. All this can do is look for keywords.
For general output colorization needs, I added the grc plugin to Tackle for precisely that purpose.
I'm new to linux and have been coding some beginenr level shell scripts.
What I want to do is write 2 scripts. The first script will read input from user and the 2nd script will display this input in a loop till it detects an "exit" from the user.
This is how I've coded the 2 shell scripts.
File1.sh:
read var1
echo $var1
File2.sh:
while [ "$var2" != "exit" ]
do
echo $1
read var2
done
Now, I want to use a named pipe to pass the output of File1.sh as input to var1 of File2.sh. I probably will have to modify code in File2.sh so that it will accept argument from a named pipe (as in instead of $1 the input will be from the named pipe), but I'm not at all sure how to go about it.
Giving the output of File1.sh as input to the named pipe can be given as follows:
mkfifo pipe
./File1.sh > pipe
This command keeps asking for input until i break out using ctrl + c. I don't know why that is.
Also how do I make the File2.sh read from this pipe?
will this be correct?
pipe|./File2.sh
I'm very new to linux but I've searched quite a lot online and there isn't even one example of doing this in shell script.
As for your original question, the syntax to read from a named pipe (or any other object in the file system) is
./File2.sh <pipe
Also, your script needs to echo "$var2" with the correct variable name, and double quotes to guard the value against wildcard expansion, variable substitution, etc. See also When to wrap quotes around a shell variable?
The code in your own answer has several new problems.
In File1.sh, you are apparently attempting to declare a variable pipe1, but the assignment syntax is wrong: You cannot have whitespace around the equals sign. Because you never use this variable for anything, this is by and large harmless (but will result in pipe1: command not found which is annoying, of course).
In File2.sh, the while loop's syntax is hopelessly screwed; you dropped the read; the echo still lacks quotes around the variable; and you repeatedly reopen the pipe.
while [ "$input" != "exit" ]
do
read -r input
echo "$input"
done <pipe1
Redirecting the entire loop once is going to be significantly more efficient.
Notice also the option -r to prevent read from performing any parsing of the values it reads. (The ugly default behavior is legacy from the olden days, and cannot be fixed without breaking existing scripts, unfortunately.)
First in File1.sh, echo var1 should be echo $var1.
In order to get input from pipe, try:
./File2.sh < pipe
This is how I solved it.
First mistake I made was to declare the pipe outside the programs. What I was expecting was there is a special way in which a program accepts input parameters of the type "pipe". Which as far as I've figured is wrong.
What you need to do is declare the pipe inside the program. So in the read program what you do is,
For File1.sh:
pipe1=/Documents
mkfifo pipe1
cat > pipe1
This will send the read input from the user to the pipe.
Now, when the pipe is open, it will keep accepting input. You can read from the pipe only when its open. So you need to open a 2nd terminal window to run the 2nd program.
For File2.sh:
while("$input" != "exit")
do
read -r input < pipe1
echo "$input"
done
So whenever you input some string in the first terminal window, it will be reflected in the 2nd terminal window until "exit" is detected.
Edit: This question was originally bash specific. I'd still rather have a bash solution, but if there's a good way to do this in another shell then that would be useful to know as well!
Okay, top level description of the problem. I would like to be able to add a hook to bash such that, when a user enters, for example $cat foo | sort -n | less, this is intercepted and translated into wrapper 'cat foo | sort -n | less'. I've seen ways to run commands before and after each command (using DEBUG traps or PROMPT_COMMAND or similar), but nothing about how to intercept each command and allow it to be handled by another process. Is there a way to do this?
For an explanation of why I'd like to do this, in case people have other suggestions of ways to approach it:
Tools like script let you log everything you do in a terminal to a log (as, to an extent, does bash history). However, they don't do it very well - script mixes input with output into one big string and gets confused with applications such as vi which take over the screen, history only gives you the raw commands being typed in, and neither of them work well if you have commands being entered into multiple terminals at the same time. What I would like to do is capture much richer information - as an example, the command, the time it executed, the time it completed, the exit status, the first few lines of stdin and stdout. I'd also prefer to send this to a listening daemon somewhere which could happily multiplex multiple terminals. The easy way to do this is to pass the command to another program which can exec a shell to handle the command as a subprocess whilst getting handles to stdin, stdout, exit status etc. One could write a shell to do this, but you'd lose much of the functionality already in bash, which would be annoying.
The motivation for this comes from trying to make sense of exploratory data analysis like procedures after the fact. With richer information like this, it would be possible to generate decent reporting on what happened, squashing multiple invocations of one command into one where the first few gave non-zero exits, asking where files came from by searching for everything that touched the file, etc etc.
Run this bash script:
#!/bin/bash
while read -e line
do
wrapper "$line"
done
In its simplest form, wrapper could consist of eval "$LINE". You mentioned wanting to have timings, so maybe instead have time eval "$line". You wanted to capture exit status, so this should be followed by the line save=$?. And, you wanted to capture the first few lines of stdout, so some redirecting is in order. And so on.
MORE: Jo So suggests that handling for multiple-line bash commands be included. In its simplest form, if eval returns with "syntax error: unexpected end of file", then you want to prompt for another line of input before proceeding. Better yet, to check for proper bash commands, run bash -n <<<"$line" before you do the eval. If bash -n reports the end-of-line error, then prompt for more input to add to `$line'. And so on.
Binfmt_misc comes to mind. The Linux kernel has a capability to allow arbitrary executable file formats to be recognized and passed to user application.
You could use this capability to register your wrapper but instead of handling arbitrary executable, it should handle all executable.
So I have this existing command that accepts a single argument, but I need something that accepts the argument over stdin instead.
A shell script wrapper like the following works, but as I will be allowing untrusted users to pass arbitrary strings on stdin, I'm wondering if there's potential for someone to execute arbitary commands on the shell.
#!/bin/sh
$CMD "`cat`"
Obviously if $CMD has a vulnerability in the way it processes the argument there's nothing I can do, so I'm concerned stuff like this:
Somehow allow the user to escape the double quotes and pass input into argument #2 of $CMD
Somehow cause another arbitary command to run
The parameter looks fine to me, but the command might be a bit shaky, if it can have a space in it. Also, if you're looking to get just one line from the user then you might prefer this:
#!/bin/bash
read line
exec "$CMD" "$line"
A lot of code would be broken if "$(cmd)" could expand to multiple words.
Mostly, I will not use | less for each and every command from the shell.
Pipe to less is used only when I actually run the command without is and find out that it does not fit on the page. That costs me two runs of the same shell command.
Is there a way so that every time a command result is more than a display page, it automatically gets piped to less?
Pipe it to less -F aka --quit-if-one-screen:
Causes less to automatically exit if the entire file can be dis-
played on the first screen.
The most significant problem with trying to do that is how to get it to turn off when running programs that need a tty.
What I would recommend is that, for programs and utilities you frequently use, create shell functions that wrap them and pipe to less -F. In some cases, you can name the function the same as the program and it will take precedence, but can be overridden.
Here is an example wrapper function which would need testing and perhaps some additional code to handle edge cases, etc.
#!/bin/bash
foo () {
if [[ -p /dev/stdout ]] # you don't want to pipe to less if you're piping to something else
then
command foo "$#" | less -F
else
command foo "$#"
fi
}
If you use the same name as I have in the example, it could break things that expect different behavior. To override the function to run the underlying program directly precede it with command:
command foo
will run foo without using the function of the same name.
You could always pipe to less -E (this will cause less to automatically quit at the end of the file). For commands with short output it would do what you want. I don't think you can automatically pipe to less when there is a lot of output.
In general, automatically piping to less requires the shell to be prescient about the output that will be produced by the commands it runs - and it is hard enough for humans to predict that without trying to make programs do so.
You could write a shell that does it for you - that captures the output (but what about stderr?) and paginates if necessary, but it would most certainly not be a standard shell.
I wrote this wrapper function and put it in my .profile. You can use this before a command and it will automatically pipe it to less if it is longer than 1 page.
lcmd ()
{
echo "$("$#")" | less -F;
};
So 'lcmd ls' would ls the current directory and pipe that output to less.