I read this example in a book:
cp /bin/cat proj33
echo -n x | ./proj33 - pipe33a > pipe33b &
./proj33 <pipe33b >pipe33a &
What do the symbols -, > and < mean?
The > is I/O redirection; it sends the standard output of the command (./proj33) to the file pipe33b.
The < symbol is also I/O redirection; it sends the standard input of the command (./proj33 again) from the file pipe33b.
The - is just an argument. It is often treated by commands as an indication to read standard input instead of a file. In this context, it is likely that proj33 normally requires a file as its first argument, but it reads from the pipe when the argument is -. Sometimes, the - is used to indicate standard output. In extreme cases, you can get one - indicating standard input and one indicating standard output on a single command. With GNU tar, you could have tar -c -f - -T - with the output being written to standard output (-f -) and the list of files to be archived read from standard input (-T -).
If you see --, that is very different; it indicates the 'end of options' for the command; anything that follows is not an option, even if it starts with a dash -.
Related
I am quite new to bash and trying to type some text into multiple files with a single command using brace expansion.
I tried: cat > file_{1..100} to write into 100 files some text that I will type in the terminal. I get the following error:
bash: file_{1..100}: ambiguous redirect
I also tried: cat > "file_{1..100}" but that creates a singe file named: file_{1..100}.
I tried: cat > `file_{1..100}` but that gives the error:
file_1: command not found
How can I achieve this using brace expansion? Maybe there are other ways using other utilities and/or pipelines. But I want to know if that is possible using only simple brace expansion or not.
You can't do this with cat alone. It only writes its output to its standard output, and that single file descriptor can only be associated with a single file.
You can however do it with tee file_{1..100}.
You may wish to consider using tee file_{01..100} instead, so that the filenames are zero-padded to all have the same width: file_001, file_002, ... This has the advantage that lexicographic order will agree with numerical order, and so ls, *, etc, will process them in numerical order. Without this, you have the situation that file_2 comes after file_10 in lexicographic order.
target could be only a pipe, not a multiple files.
If you want redirect output to multiple files, use tee
cat | tee file_{1..100}
Don't forget to check man tee, for example if you want to append to the files, you should add -a option (tee -a file_{1..100})
This types the string or text into file{1..4}
echo "hello you just knew me by kruz" > file{1..4}
Use to remove them
rm file*
I'm trying to display the output of an AWS lambda that is being captured in a temporary text file, and I want to remove that file as I display its contents. Right now I'm doing:
... && cat output.json && rm output.json
Is there a clever way to combine those last two commands into one command? My goal is to make the full combined command string as short as possible.
For cases where
it is possible to control the name of the temporary text file.
If file is not used by other code
Possible to pass "/dev/stdout" as the.name of the output
Regarding portability: see stack exchange how portable ... /dev/stdout
POSIX 7 says they are extensions.
Base Definitions,
Section 2.1.1 Requirements:
The system may provide non-standard extensions. These are features not required by POSIX.1-2008 and may include, but are not limited to:
[...]
• Additional character special files with special properties (for example, /dev/stdin, /dev/stdout, and /dev/stderr)
Using the mandatory supported /dev/tty will force output into “current” terminal, making it impossible to pipe the output of the whole command into different program (or log file), or to use the program when there is no connected terminals (cron job, or other automation tools)
No, you cannot easily remove the lines of a file while displaying them. It would be highly inefficient as it would require removing characters from the beginning of a file each time you read a line. Current filesystems are pretty good at truncating lines at the end of a file, but not at the beginning.
A simple but extremely slow method would look like this:
while [ -s output.json ]
do
head -1 output.json
sed -i 1d output.json
done
While this algorithm is plain and simple, you should know that each time you remove the first line with sed -i 1d it will copy the whole content of the file but the first line into a temporary file, resulting in approximately 0.5*n² lines written in total (where n is the number of lines in your file).
In theory you could avoid this by do something like that:
while [ -s output.json ]
do
line=$(head -1 output.json)
printf -- '%s\n' "$line"
fallocate -c -o 0 -l $((${#len}+1)) output.json
done
But this does not account for variable newline characters (namely DOS-formatted newlines) and fallocate does not always work on xfs, among other issues.
Since you are trying to consume a file alongside its creation without leaving a trace of its existence on disk, you are essentially asking for a pipe functionality. In my opinion you should look into how your output.json file is produced and hopefully you can pipe it to a script of your own.
command 'which' shows the link to a command.
command 'less' open the file.
How can I 'less' the file as the output of 'which'?
I don't want to use two commands like below to do it.
=>which script
/file/to/script/fiel
=>less /file/to/script/fiel
This is a use case for command substitution:
less -- "$(which commandname)"
That said, if your shell is bash, consider using type -P instead, which (unlike the external command which) is built into the shell:
less -- "$(type -P commandname)"
Note the quotes: These are important for reliable operation. Without them, the command may not work correctly if the filename contains characters inside IFS (by default, whitespace) or can be evaluated as a glob expression.
The double dashes are likewise there for correctness: Any argument after them is treated as positional (as per POSIX Utility Syntax Guidelines), so even if a filename starting with a dash were to be returned (however unlikely this may be), it ensures that less treats that as a filename rather than as the beginning of a sequence of options or flags.
You may also wish to consider honoring the user's pager selection via the environment variable $PAGER, and using type without -P to look for aliases, shell functions and builtins:
cmdsource() {
local sourcefile
if sourcefile="$(type -P -- "$1")"; then
"${PAGER:-less}" -- "$sourcefile"
else
echo "Unable to find source for $1" >&2
echo "...checking for a shell builtin:" >&2
type -- "$1"
fi
}
This defines a function you can run:
cmdsource commandname
You should be able to just pipe it over, try this:
which script | less
I want my script to read a string either from stdin , if it's piped, or from an argument. So first i want to check if some text is piped and if not it should use an argument as input. My code looks something like this:
value=$(cat) # read from stdin
if [ "$pipe" != "" ]; then #check if pipe is not empty
#Do something with pipe string
else
#Do something with argument string
fi
The problem is when it's not piped, then the script will halt and wait for "ctrl d" and i dont want that. Any suggestions on how to solve this?
Thanks in advance.
/Tomas
What about checking the argument first?
if (($#)) ; then
process "$1"
else
cat | process
fi
Or, just take advantage from the same behaviour of cat:
cat "$#" | process
If you only need to know if it's a pipe or a redirection, it should be sufficient to determine if stdin is a terminal or not:
if [ -t 0 ]; then
# stdin is a tty: process command line
else
# stdin is not a tty: process standard input
fi
[ (aka test) with -t is equivalent to the libc isatty() function.
The above will work with both something | myscript and myscript < infile. This is the simplest solution, assuming your script is for interactive use.
The [ command is a builtin in bash and some other shells, and since [/test with -tis in POSIX, it's portable too (not relying on Linux, bash, or GNU utility features).
There's one edge case, test -t also returns false if the file descriptor is invalid, but it would take some slight adversity to arrange that. test -e will detect this, though assuming you have a filename such as /dev/stdin to use.
The POSIX tty command can also be used, and handles the adversity above. It will print the tty device name and return 0 if stdin is a terminal, and will print "not a tty" and return 1 in any other case:
if tty >/dev/null ; then
# stdin is a tty: process command line
else
# stdin is not a tty: process standard input
fi
(with GNU tty, you can use tty -s for silent operation)
A less portable way, though certainly acceptable on a typical Linux, is to use GNU stat with its %F format specifier, this returns the text "character special file", "fifo" and "regular file" in the cases of terminal, pipe and redirection respectively. stat requires a filename, so you must provide a specially-named file of the form /dev/stdin, /dev/fd/0, or /proc/self/fd/0, and use -L to chase symlinks:
stat -L -c "%F" /dev/stdin
This is probably the best way to handle non-interactive use (since you can't make assumptions about terminals then), or to detect an actual pipe (FIFO) distinct from redirection.
There is a slight gotcha with %F in that you cannot use it to tell the difference between a terminal and certain other device files, for example /dev/zero or /dev/null which are also "character special files" and might reasonably appear. An unpretty solution is to use %t to report the underlying device type (major, in hex), assuming you know what the underlying tty device number ranges are... and that depends on whether you're using BSD style ptys or Unix98 ptys, or whether you're on the actual console, among other things. In the simple case %t will be 0 though for a pipe or a redirection of a normal (non-special) file.
More general solutions to this kind of problem are to use bash's read with a timeout (read -t 0 ...) or non-blocking I/O with GNU dd (dd iflag=nonblock).
The latter will allow you to detect lack of input on stdin, dd will return an exit code of 1 if there is nothing ready to read. However, these are more suitable for non-blocking polling loops, rather than a once-off check: there is a race condition when you start two or more processes in a pipeline as one may be ready to read before another has written.
It's easier to check for command line arguments first and fallback to stdin if no arguments. Shell Parameter Expansion is a nice shorthand instead of the if-else:
value=${*:-`cat`}
# do something with $value
I see the - operator behaving in different ways with different commands.
For example,
cd -
cds to the previous directory, whereas,
vim -
reads from stdin
So I want to know why the - operator is behaving in 2 different ways here. Can someone point me to some detailed documentation of the - operator?
It is not an operator, it is an argument. When you write a program in C or C++ it comes as argv[1] (when it is the first argument) and you can do whatever you like with it.
By convention, many programs use - as a placeholder for stdin where an input file name is normally required, and stdout where an output file name is expected. But cd does not require reading a file stream, why should it need stdin or stdout?
Extra: here below is the excerpt from vim's main.c that parses arguments that begin with -: if there is no additional character it activates STDIN input.
else if (argv[0][0] == '-' && !had_minmin)
{
want_argument = FALSE;
c = argv[0][argv_idx++];
#ifdef VMS
...
#endif
switch (c)
{
case NUL: /* "vim -" read from stdin */
/* "ex -" silent mode */
if (exmode_active)
silent_mode = TRUE;
else
{
if (parmp->edit_type != EDIT_NONE)
mainerr(ME_TOO_MANY_ARGS, (char_u *)argv[0]);
parmp->edit_type = EDIT_STDIN;
read_cmd_fd = 2; /* read from stderr instead of stdin */
}
The dash on its own is a simple command argument. Its meaning is command dependent. Its two most usual meanings are 'standard input' or (less often) 'standard output'. The meaning of 'previous directory' is unique to the cd shell built-in (and it only means that in some shells, not all shells).
cat file1 - file2 | troff ...
This means read file1, standard input, and file2 in that sequence and send the output to troff.
An extreme case of using - to mean 'standard input' or 'standard output' comes from (GNU) tar:
generate_file_list ... |
tar -cf - -T - |
( cd /some/where/else; tar -xf - )
The -cf - options in the first tar mean 'create an archive' and 'the output file is standard output'; the -T - option means 'read the list of files and/or directories from standard input'.
The -xf - options in the second tar mean 'extract an archive' and 'the input file is standard input'. In fact, GNU tar has an option -C /some/where/else which means it does the cd itself, so the whole command could be:
generate_file_list ... |
tar -cf - -T - |
tar -xf - -C /some/where/else
The net effect of this is to copy the files named by the generate_file_list command from under the 'current directory' to /some/where/else, preserving the directory structure. (The 'current directory' has to be taken with a small pinch of salt; any absolute file names are given special treatment by GNU tar — it removes the leading slash — and relative names are taken as relative to the current directory.)
It depends on the program it's being used in. It means different things to different programs.
I think different program use different convention. manpages shows how each program interpret -, here is man bash
-
At shell startup, set to the absolute pathname used to invoke the shell
or shell script being executed as passed in the environment or argument list.
Subsequently, expands to the last argument to the previous command, after expansion.
Also set to the full pathname used to invoke each command executed and placed
in the environment exported to that command. When checking mail, this parameter
holds the name of the mail file currently being checked.
and man vim
- The file to edit is read from stdin. Commands are read from stderr,
which should be a tty.