I'm trying to get every new line with tail from a file and transform it to hexdump, but I'm unable to write it to a file, I've tried with >> and | tee -a destfile but it doesn't give any error but stops working.
So, I have a binary file (data.bin) that is always growing new lines with a script.I'm trying to read and transform it to hexadecimal, that works well, and it outputs well:
tail -f data.bin | hexdump -e '16/1 "%02x " "\n"'
this outputs this:
01 55 1d fa 14 ae b5 41 ec 51 3c 42 64 55 00 00
74 5e f7 5d 00 00 00 00 02 55 1d fa 33 33 b3 41
7b 14 3f 42 63 55 00 00 74 5e f7 5d 00 00 00 00
When I try to do this
tail -f data.bin | hexdump -e '16/1 "%02x " "\n"' >> destfile.txt
It creates an empty file and doesn't write anything.
When you redirect the output of a command, the output stream becomes fully buffered, so it will not be visible until flushed. You could read for exaple this link for more information.
To make the output line buffered you can use stdbuf utility from GNU coreutils:
stdbuf -oL tail -f input | stdbuf -oL hexdump -e '16/1 "%02x " "\n"' >> destfile.txt
That way the output will be flushed each line.
Related
I have a file which contains several paragraphs as below:
[730480.910190] [MACSTATUSIND] ACTIND_ParseMACSTATUS:
[730480.910205] fe 0a 39 01 0a 00 51 e7 ba 9d c7 0d 00 00 00 00 ..9...Q....
[730480.910220] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...........
[730480.910233] 00 00 00 00 00 00 00 00 00 00 xx xx xx xx xx xx .....:bC.B.
[730480.910247] xx xx yy yy yy yy yy yy zz zz 64 34 e8 ff 00 00 ......d4...
from above data, for each paragraph i want to make three columns:
xxxxxxxx | yyyyyy | zzzz
| |
| |
and then store those rows in a different/new file.
I tried using grep and cut, but it's too lengthy and don't have enough idea about sed & awk commands.
How to solve above scenario using sed and awk commands!!
Consider this a 'code golf' solution, at least I had some fun with it;)
<file sed -n '/MACSTAT/ {n;n;n;p;n;p}' |
cut -c 17-63 |
paste -d' ' - - |
cut -c 31-78 |
sed -e 's/ /|/8' -e 's/ /|/13' |
tr -d ' '
I suggest to remove the parts of the pipe from the end to figure out what exactly the commands do to the input. In general the first sed finds the header, skips three lines, prints the next two. Then cut keeps only the hexprint. Then paste joins each two consecutive lines into one. Then cut takes out only the required bytes. Then sed substitutes particular spaces with column separators. Then tr removes the remaining spaces.
You can try this sed (GNU sed) 4.4
sed -nE '
/\[MACSTATUSIND] ACTIND_ParseMACSTATUS:/N;N;N;N
# get 5 lines if the first contain MACSTAT...
s/.*\n[^ ]* //
# keep only the last line after the first space
s/ //g
# remove all spaces
s/(.{4})(.{12})(.{4}).*/\1|\2|\3/p
# keep only the 20 char you want and
# format them in 3 columns with delimiter |
' infile
I am executing following command successfully from linux env,
[UdayaShankar Das#lxapp-3]# tshark -r sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*\s\(\(\s[0-9a-f][0-9a-f]\)\{1,16\}\).*$/\1/p'
ff ff ff ff ff ff 00 10 94 00 00 0a 08 06 00 01
08 00 06 04 00 01 00 10 94 00 00 0a c0 01 01 02
00 00 00 00 00 00 c0 01 01 01 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bu the above command failing when trying from tcl shell through exec command and back tik operator . Please advise.
% [ exec tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p' ]
invalid command name "0-9a-f"
UPDATE:
thanks for your suggestion. I tried but seems not working :(
% set cmd34 {tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p'}
tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p'
% [ eval exec $cmd34]
invalid command name "0-9a-f"
Also tried with second option
%set cmd34 {tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p'}
tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p'
% [ exec {*}$cmd34 ]
extra characters after close-brace
When you have a chunk of shell script like this:
tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p'
The simplest method of running it is to use:
set shellscript {tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p'}
set response [exec /bin/sh -c $shellscript]
Alternatively, change the single quote characters to braces:
exec tshark -r /home/arts2/STC_CAPTURE/sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n {s/^[0-9a-f]*s((s[0-9a-f][0-9a-f]){1,16}).*^A/p}
You can set the command into a variable and use {*} or eval to evaluate it.
set cmd {tshark -r sim_venom_sanity_DEVICE1_PORT1_1_capture.pcap -Vx -R frame.number==2 | sed -n 's/^[0-9a-f]*\s\(\(\s[0-9a-f][0-9a-f]\)\{1,16\}\).*$/\1/p'}
puts [ eval exec $cmd ]
If you have a tcl 8.5 or higher , you can use the below code, instead of eval
puts [ exec {*}$cmd ] ; # {*} is for argument expansion which is recommended
Please note that the expression is enclosed with braces to avoid any substitutions and treat them as literal values. If you need to use braces in the expression, make sure you are escaping it with backslash like \{
I am trying to convert a raw file (header-less) to wav
It is in ADPCM 16 rate 8000
I have tried three console commands
sox -e ima-adpcm -r 8000 input.raw output.wav
sox -e ima-adpcm -r 8000 -b 16 input.raw output.wav
These trigger the same error:
sox FAIL formats: bad input format for file `input.raw': data encoding or sample size was not specified
and last attempt
sox -t raw -t wav -e ima-adpcm -r 8000 -b 16 input.raw output.wav
which triggers another error
sox FAIL formats: can't open input file `input.raw': WAVE: RIFF header not found
Do you know why I get these errors?
Thanks
EDIT
An hex dump of the first 48 bytes FYI
C0 0C 00 03 00 00 37 C0 34 0C D0 30 37 C0 30 C7 CD D3 C0 03 CD 00 37 03
C0 00 0D 0C CD 00 C0 10 0F 00 03 55 0B 4B 0F 0F 0C 0F 34 0F 0D C3 0D 03
00 C0 31 00 00 D0 C0 00 F4 4C 03 33 34 33 31 33 0D 33 D0 03 C0 4C C0 43
Looks like using -t raw -e ima-adpcm doesn't work, but rather you have to use -t ima -e ima-adpcm.
I guess -t raw means raw PCM, whereas -t ima means raw ADPCM?
This worked for me to convert an ADPCM file into .wav:
sox -t ima -r 44100 -e ima-adpcm in.raw -e signed-integer -b 16 out.wav
The file extension doesn't seem to matter as long as you indicate ADPCM with -t ima.
Can you change the filename from input.raw to input.ima ? If you can, then
sox input.ima -r 8000 -b 16 output.wav
ought to work. .raw seems to be a magic extension for sox and it assumes certain possibilities for the data. See this man page.
I have a problem with my website, which is completely gzipped:
http://goout.cz/cs/fotoreporty/
The page can be easily shown in Chrome, but in Safari it never loads (and I suppose in other browsers as well. When I try:
curl -v http://goout.cz/cs/fotoreporty/ | gzip -d
I am getting expected results. But validation on
http://validator.w3.org/check?uri=http%3A%2F%2Fgoout.cz%2Fcs%2Ffotoreporty%2F#fatal-errors
yields:
The error was: Can't gunzip content: Header Error: Minimum header size is 10 bytes
What is wrong with the gzip format? How can I solve it? Thanks.
EDIT:
The gzip header seems to me okay to me:
$ curl http://goout.cz/cs/ | head -1 | hexdump | head -1
0000000 1f 8b 08 00 00 00 00 00 00 00 ed 5d cd 73 db 38
$ curl http://goout.cz/cs/fotoreporty/ | head -1 | hexdump | head -1
0000000 1f 8b 08 00 00 00 00 00 00 00 ed 7d cf 73 e3 46
Although it is looking similar to my previous post but here purpose is different.
udit#udit-Dabba ~/ah $ cat decrypt.txt
60 00 00 00 00 17 3a 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 *00 00 e0 f9 6a 61 61 6e
65 6b 61 68 61 6e 67 61 79 65 77 6f 64 69 6e* 00
00 00 03 29
I want to extract last string of the file (here it is 29) in a shell varaible
I tried this ...
size=`wc -w encrypt.txt`
awk -v size=$size 'BEGIN {RS=" ";ORS=" ";}' {if (NR>size-1 &&
NR < size+1)print $0}' decrypt.txt
Output :
29
But when I changed the file slightly ..
udit#udit-Dabba ~/ah $ cat decrypt.txt
60 00 00 00 00 17 3a 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 *00 00 e0 f9 6a 61 61 6e
65 6b 61 68 61 6e 67 61 79 65 77 6f 64 69 6e* 00
65 6b 61 68 61 6e 67 61 00 00 03 29
Output :
03
Why there is discrepency between the results ??
I am new to awk and shell features so I am not sure whether it is a right way to do so or not ???
I think there should be some variation of grep,sed,awk or any other linux command which may solve my problem but I am not aware of it.
Please guide me for this.
Thanx in advance.
Purpose :
Make two variables in a shell script which should store last and second last strings of an input file.
Limitation :
Every input file contains a blank line at the end of file.
(Like in above mentioned file , after the file contents there would be one more blank line just like hitting ENTER key and that can not be changed because it is being generated through a C program at run time.)
grep -v "^$" file | tr " " "\n" | tail -n 2
Maybe the grep-part isn't perfect and maybe should change.
Edit
tr -s " " "\n" < file | tail -n 2
is better solution - see Gordon Davisson's comment.
To get the last field:
awk '{ if (NF > 0) { last = $NF } } END { print last }' "$#"
The second last field is trickier for the case where there is just one field on the last line (so you need the last field from the line before).
awk '{ if (NF > 0)
{
if (NF == 1) { lastbut1 = last; last = $1; }
else { lastbut1 = $(NF-1); last = $NF; }
}
}
END { print lastbut1 " " last; }' "$#"
This produces a blank and the last value if the file contains but one value. It produces just a blank if there are no values at all.
If you consider the record separator to be space or newline, then you just need to keep the last 2 records.
awk -v 'RS=[ \n]+' '{prev2 = prev1; prev1 = $0} END {print prev2, prev1}' filename
FIRST="$(head -n 1 file)"
LAST="$(tail -n 1 file)"
LASTBUTONE="$(tail -n 2 file | head -n 1)"
naturally, you can cut off the last field in a variety of ways:
echo "$ONEOFTHOSE" | gawk '{print $(NF)}'
echo "$ONEOFTHOSE" | sed -e 's/^.*[[:space:]]//'
Here's a tr/sed solution:
answers=$(tr -d '\n' <input_file | sed -r 's/.*(\S\S)\s*(\S\S)\s*$/\1 \2/')
echo "Last = ${answers#???} Penultimate = ${answers%???}"
Sed only:
answers=$(sed -r '1{h;d};H;${x;y/\n/ /;s/.*(\S\S)\s*(\S\S)\s*$/\1 \2/p};d' input_file)
echo "Last = ${answers#???} Penultimate = ${answers%???}"
If you got 'rev' utility installed below one would be handy. Presuming that space is the delimiter.
rev <file>|cut -f1,2|rev