Why is sort -c output not being redirected to file temp.txt?
If I remove -c, it does redirect, as seen below:
$ cat numericSort
33:Thirty Three:6
11:Eleven:2
45:Forty Five:9
01:Zero One:1
99:Ninety Nine:9
18:Eighteen:01
56:Fifty Six:4
78:Seventy Eight:2
$ sort numericSort > temp.txt
$ cat temp.txt
01:Zero One:1
11:Eleven:2
18:Eighteen:01
33:Thirty Three:6
45:Forty Five:9
56:Fifty Six:4
78:Seventy Eight:2
99:Ninety Nine:9
$ rm temp.txt
$ sort -c numericSort > temp.txt
sort: numericSort:2: disorder: 11:Eleven:2
$ cat temp.txt
# No Output Here
The output of sort -c goes to stderr, not stdout.
If you want to redirect that instead:
$ sort -c numericSort 2> temp.txt
Based on document of sort command
-c, --check, --check=diagnose-first
check for sorted input; do not sort
Check whether the given files are already sorted: if they are not
all sorted, print an error message and exit with a status of 1.
so your command sort -c numericSort > temp.txt is essentially just checking whether the file numericSort is sorted or not. If not sorted print error on STDERR and hence you don't see any output to temp.txt. Probably you want to redirect STDERR instead like
sort -c numericSort 2> temp.txt
Related
Doing the following:
First console
touch /tmp/test
Second console
tail -f /tmp/test |grep propo |grep -v miles
Third console
echo propo >> /tmp/test
Second console must show "propo" but it doesn't shows anything, if you run in second console instead:
tail -f /tmp/test |grep propo
And do echo propo >> /tmp/test it will show propo, but the grep -v is for miles not for propo
Why?
Test into your own environment if you want, it's pretty obvious but not working.
Why?
Most probably because the output of a command when piped to another command is fully buffered, not line buffered. The output could be buffered in the first pipe or by grep.
Use stdbuf -oL to force line buffering and grep --line-buffered for line buffered grep.
the problem is that grep does not use line buffering by default; so the output will be buffered. You could use grep --line-buffered:
tail -f /tmp/test | grep --line-buffered propo | grep -v miles
I run some command that generates ip address as output. But I am developing a workflow where I need the ip address to be written twice. Below is the sample command and its output.
$ some command >> out.txt
$ cat out.txt
10.241.1.85
hdfs://10.241.1.236/
hdfs://10.241.1.237/
What i want is to duplicate the output and it should look like this.
10.241.1.85
hdfs://10.241.1.236/
hdfs://10.241.1.237/
10.241.1.85
hdfs://10.241.1.236/
hdfs://10.241.1.237/
Any help please?
The solution given by #ott in a comment seems fine:
var=$(some cmd); echo -e "$var\n$var".
This is not assigning the command o a varable but it is assigning the output of the command to a variable.
When you do not want this, you can use tee (perhaps this will give some ordering problems) or duplicate it differently:
some_command > out.txt.tmp
cat out.txt.tmp out.txt.tmp > out.txt
rm out.txt.tmp
This way you first get the lines of the copy after al the lines of first entries. When you want to double the output directly, you can use
some_command | sed 'p' > out.txt
some command | tee -a out.txt out.txt
Or
some command | tee -a out.txt >> out.txt
Or
some command | tee -a out.txt out.txt >/dev/null
Run command
Pipe to tee
Enable append mode
Append to same file twice
Generate the output to some temporary, then duplicate the temporary to destination and remove temporary :
some command > /tmp/out.txt; cat /tmp/out.txt /tmp/out.txt > out.txt; rm /tmp/out.txt
Here are some more options you could play around with. Seeing as the output is too large to store in a variable, I'd probably go with tee, a temp file, and gzip if the disk write speed is a bottleneck.
someCommand > tmp.txt && cat tmp.txt tmp.txt > out.txt && rm tmp.txt
Now, if the disk read/write speed is a bottleneck, you can tee the output of someCommand and redirect one of the pipelines through gzip initially.
someCommand | tee >(gzip > tmp.gz) > out.txt && gunzip -c tmp.gz >> out.txt && rm tmp.gz
Additionally, if you don't need random access abilities for out.txt and plan on processing it through some other pipeline, you could always keep it stored gzipped until you need it.
someCommand | gzip > tmp.gz && cat tmp.gz tmp.gz > out.txt.gz && rm tmp.gz
I would suggest this:
(someCommand | tee tmp.txt; cat tmp.txt) > out.txt; rm tmp.txt
Not sure there's a way to safely do this without resorting to a temporary file. You could capture it to a variable, as some have suggested, but you have to be careful about quoting then to make sure whitespace doesn't get mangled, and you also might run into problems if the output is particularly large.
I want to send a program the most recent lines from a text file using tail as stdin.
First, I echo to the program some input that will be the same every time, then send in tail input from an inputfile which should first be processed through sed. The following is the command line that I expect to work. But when the program runs it only receives the echo input, not the tail input.
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program
However, the following works exactly as expected, printing everything out to the terminal:
echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat
So I tried with another type of output, and again while the echoed text posted, the tail text does not appear anywhere:
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') | tee out.txt
This made me think it is a problem with buffering, but I tried the unbuffer program and all other advice here (https://superuser.com/questions/59497/writing-tail-f-output-to-another-file) without results. Where is the tail output going and how can I get it to go into my program as expected?
The buffering problem was resolved when I prefixed the sed command with the following:
stdbuf -i0 -o0 -e0
Much more preferable to using unbuffer, which didn't even work for me. Dave M's suggestion of using sed's relatively new -u also seems to do the trick.
One thing you may be getting confused by -- | (pipeline) is higher precedence than && (consecutive execution). So when you say
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program
that is equivalent to
(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') && cat) | ./program
So the cat isn't really doing anything, and the sed output is probably buffered a bit. You can try using the -u option to sed to get it to use unbuffered output:
(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -u -r 'some regex')) | ./program
I believe some versions of sed default to -u when the output is a terminal and not when it is a pipe, so that may be the source of the difference you're seeing.
You can use the i command in sed (see the command list in the manpage for details) to do the inserting at the beginning:
tail -f inputfile | sed -e '1inew file' -e 's/this/that/' | ./program
I am trying to create a sh that generates a report that will display differences against files from two folders/java projects. I am using dwdiff, and I need only the last line from each comparison (I don't care about differences, the percentage is what I need). I've created the following script:
DEST_FOLDER='./target/'
CLASSES_P1='classesP1.txt'
CLASSES_P2='classesP2.txt'
DEST_FILE='report.txt'
rm -r "$DEST_FOLDER"
mkdir -p "$DEST_FOLDER"
rm -f "$DEST_FOLDER/$CLASSES_P1"
rm -f "$DEST_FOLDER/$CLASSES_P2"
rm -f "$DEST_FOLDER/$DEST_FILE"
find ./p1 -name "*.java" >> "$DEST_FOLDER/$CLASSES_P1"
find ./p2 -name "*.java" >> "$DEST_FOLDER/$CLASSES_P2"
while read p; do
while read t; do
dwdiff -s $p $t | tail --lines=0 >> "$DEST_FOLDER/$DEST_FILE"
done < $DEST_FOLDER/$CLASSES_P2
done < $DEST_FOLDER/$CLASSES_P1
It works fine, but the results are not redirected to the given file. The file is created, but is empty, and the last line from each dwdiff result is displayed to console. Any ideas?
There are a few things going on
The output you want is going to stderr, not stdout. You can merge them with 2>&1 on the dwdiff command
The output you want actually appears to be printed first by dwdiff, but you see a different order due to the two different outputs. So you want head instead of tail
You want 1 line, not 0
So try dwdiff -s ... 2>&1 | head --lines=1
$ dwdiff -s /etc/motd /etc/issue 2>&1 | head --lines=1
old: 67 words 3 4% common 2 2% deleted 62 92% changed
Alternatively, if you want the new line instead of the old, and to simplify the ordering, try throwing away the diff output:
$ dwdiff -s /etc/motd /etc/issue 2>&1 1>/dev/null | tail --lines=1
new: 5 words 3 60% common 0 0% inserted 2 40% changed
Note that the order of redirection is important: First clone stdout into stderr, then redirect stdout to /dev/null.
I am trying to capture the output of "sort -c" in linux. I tried redirecting it to a file, used tee command but both did not helped. Any suggestions ?
For example:
roor>cat db | sort -c
sort: -:319: disorder: 1842251880: aa bb bc dd ee
Following failed to give me output
roor>cat db | sort -c > fileName
roor>cat db | sort -c |tee fileName
Sample file:
>cat file
111 aa as sdasd
222 sadf dzfasf af
333 sada gvsdgf h hgfhfghfg
444 asdfafasfa gsdgsdg sgsg
222 asdasd fasdfaf asdasdasd
root>cat file |sort -c
sort: -:5: disorder: 222 asdasd fasdfaf asdasdasd
8>sort -c db 2> fileName
sort: extra operand `2' not allowed with -c
0>sort -c < file 2> result1.txt
sort: open failed: 2: No such file or directory
ANY ALTERNATE TO SORT -C would ALSO WORK FOR ME!!
If sort -c is producing an error, it sends that error to "standard error" (stderr), not to "standard output" (stdout).
In shell, you need to use special redirects to capture standard error.
sort -c inputfile > /path/to/stdout.txt 2> /path/to/stderr.txt
These two output streams are called "file descriptors", and you can alternately redirect one of them to the other:
sort -c inputfile > /path/to/combined.txt 2>&1
You can read more about how these work at tldp.org, in the Bash reference manual, the bash-hackers wiki and the Bash FAQ. Happy reading! :-D
other good alternative for '2>' is STDERR pipe
|&
cat db | sort -c -h |& tee >fileName
Some time it is very suitable when present STDIN, for example:
TIMEFORMAT=%R;for i in `seq 1 20` ; do time kubectl get pods -l app=pod >/dev/null ; done |& sort -u -h
or
TIMEFORMAT=%R;for i in `seq 1 20` ; do time kubectl get pods >>log1 ; done |& sort -u -h
sort -c has no output as you might expect:
[root#home:~] cat /etc/services | sort -c
sort: -:2: disorder: #
As described by the manpage, the -c argument simply checks whether a given file or input is sorted or not.
If you're trying to catch the message from the command, try redirecting the error stream (2), not the standard output (1):
cat file | sort -c 2>result.txt
sort -conly checks if the input is sorted. It does not performs any sorting.
See: man sort
Remove -c to sort the lines.
PS: It gives the "disorder" error because the file of yours isn't already sorted. On line 5, "222" appears after "444" on the previous line.
EDIT: I think I misunderstood.
To redirect the error to a file you must use 2>.
So, the command would become: roor>cat db | sort -c 2> fileName
EDIT2: You can simply use: sort -c db 2> fileName