Creating new file with Square root of specific columns using awk - linux

I have to find the square root of values in some specific columns in the file test.text and write the resulting column into another file. The file test.txt is shown below.
0004 0015 0018 0007 0015 0009 0003 0018 0012 0007 0013 0010 0018 0015 0002 0020 0007 0005
0010 0018 0015 0018 0007 0011 0010 0004 0013 0017 0016 0010 0019 0003 0002 0016 0005 0009
0002 0011 0015 0014 0008 0017 0002 0001 0015 0004 0008 0008 0002 0011 0015 0015 0009 0011
0013 0020 0005 0007 0007 0004 0019 0011 0001 0008 0013 0015 0015 0012 0007 0019 0009 0016
0003 0003 0020 0015 0007 0017 0005 0010 0012 0012 0009 0004 0016 0006 0014 0019 0001 0001
0014 0006 0016 0003 0008 0017 0002 0001 0011 0015 0014 0008 0016 0006 0014 0019 0012 0007
I have to find the square root of values from 2nd, 5th, 8th, 11th.. etc columns and write them into another file.
After searching, I have obtained the below code to print the square root of only one column.
awk '{r=$2; print "root=" sqrt(r)}' test.txt
This code prints only square root of column 2 values.
So I tried the below code using for loop.
awk 'BEGIN{FS=" "}{for(r=2;r<=NF;r+=3) printf sqrt($r)}' test.txt >newtest.txt
But still I am not getting the desired result as separate columns. I need newtest.txt file containing columns with square root values of 2nd , 5th, 8th ..etc columns of test.txt.
Any help regarding this will be appreciated.
Edit
The sample output I expect is newtest.txt which is shown below:
3.87298334621 3.87298334621 4.24264068712 3.60555127546 3.87298334621 2.64575131106
4.24264068712 2.64575131106 2.00000000000 4.00000000000 1.73205080757 2.23606797750
3.31662479036 2.82842712475 1.00000000000 2.82842712475 3.31662479036 3.00000000000
4.47213595500 2.64575131106 3.31662479036 3.60555127546 3.46410161514 3.00000000000
1.73205080757 2.64575131106 3.16227766017 3.00000000000 2.44948974278 1.00000000000
2.44948974278 2.82842712475 1.00000000000 3.74165738677 2.44948974278 3.46410161514

awk '{for(r=2;r<=NF;r+=3) printf("%.11f ",sqrt($r)); print ""}' test.txt >newtest.txt
Output to newtest.txt:
3.87298334621 3.87298334621 4.24264068712 3.60555127546 3.87298334621 2.64575131106
4.24264068712 2.64575131106 2.00000000000 4.00000000000 1.73205080757 2.23606797750
3.31662479036 2.82842712475 1.00000000000 2.82842712475 3.31662479036 3.00000000000
4.47213595500 2.64575131106 3.31662479036 3.60555127546 3.46410161514 3.00000000000
1.73205080757 2.64575131106 3.16227766017 3.00000000000 2.44948974278 1.00000000000
2.44948974278 2.82842712475 1.00000000000 3.74165738677 2.44948974278 3.46410161514

Related

imagemagick gm node buffer file conversion

I'm currently using ImageMagick and gm to process images from a buffer. My problem is that I cannot control what file types are put in the buffer but wish to have everything turned to jpg.
Not sure how to do that using the buffer since I'm not giving it an output file path with extension.
gm(buf).command('convert').in('-auto-orient','-resize','500x','-quality','92','-strip','-quality','100','jpg').toBuffer((err, buffer) => err ? reject(err) : resolve(buffer));
I don't know anything about the node.js bindings of ImageMagick or GraphicsMagick, but I do know that if you do this in Terminal, you will force a JPEG output, so maybe you can adapt that...
# Make a 1x1 black image and write to stdout as JPEG. Dump result with 'xxd'
gm convert xc:black jpg:- | xxd
00000000: ffd8 ffe0 0010 4a46 4946 0001 0101 0048 ......JFIF.....H
00000010: 0048 0000 ffdb 0043 0008 0606 0706 0508 .H.....C........
00000020: 0707 0709 0908 0a0c 140d 0c0b 0b0c 1912 ................
00000030: 130f 141d 1a1f 1e1d 1a1c 1c20 242e 2720 ........... $.'
00000040: 222c 231c 1c28 3729 2c30 3134 3434 1f27 ",#..(7),01444.'
00000050: 393d 3832 3c2e 3334 32ff c000 0b08 0001 9=82<.342.......
00000060: 0001 0101 1100 ffc4 0014 0001 0000 0000 ................
00000070: 0000 0000 0000 0000 0000 0008 ffc4 0014 ................
00000080: 1001 0000 0000 0000 0000 0000 0000 0000 ................
00000090: 0000 ffda 0008 0101 0000 3f00 3fbf ffd9 ..........?.?...
It is just the same with ImageMagick and, say, PNG output:
magick xc:black png:-

Unix -Delete even bytes from a file

I want to find and delete all even bytes from a file given in command line. Is there any command for this situation?
I think this does what you want. It dumps the file as continuous plain hex, then reads two bytes, saving them for later and then two more bytes. Then it outputs the bytes it saved and pipes the whole lot back into xxd to "reconstruct":
xxd -p INPUT | sed -E 's/(..)../\1/g' | xxd -r -p > OUTPUT
So, if I dump a PNG image like this:
xxd image.png
00000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452 .PNG........IHDR
00000010: 0000 0064 0000 0064 0203 0000 000d 8c7d ...d...d.......}
00000020: c700 0000 0467 414d 4100 00b1 8f0b fc61 .....gAMA......a
00000030: 0500 0000 2063 4852 4d00 007a 2600 0080 .... cHRM..z&...
00000040: 8400 00fa 0000 0080 e800 0075 3000 00ea ...........u0...
00000050: 6000 003a 9800 0017 709c ba51 3c00 0000 `..:....p..Q<...
00000060: 0950 4c54 4500 0000 ff00 00ff ffff 6719 .PLTE.........g.
00000070: 641e 0000 0001 624b 4744 0266 0b7c 6400 d.....bKGD.f.|d.
00000080: 0000 0774 494d 4507 e404 1c0a 0820 506e ...tIME...... Pn
00000090: 92c5 0000 0025 4944 4154 48c7 6360 1805 .....%IDATH.c`..
000000a0: a300 0c58 4361 2064 5466 5466 5466 5466 ...XCa dTfTfTfTf
000000b0: 5466 5406 8fcc 2818 3100 00e8 0fcb 7f57 TfT...(.1......W
000000c0: afef dd00 0000 2574 4558 7464 6174 653a ......%tEXtdate:
000000d0: 6372 6561 7465 0032 3032 302d 3034 2d32 create.2020-04-2
000000e0: 3854 3130 3a30 383a 3332 2b30 303a 3030 8T10:08:32+00:00
000000f0: d513 d3d0 0000 0025 7445 5874 6461 7465 .......%tEXtdate
00000100: 3a6d 6f64 6966 7900 3230 3230 2d30 342d :modify.2020-04-
00000110: 3238 5431 303a 3038 3a33 322b 3030 3a30 28T10:08:32+00:0
00000120: 30a4 4e6b 6c00 0000 0049 454e 44ae 4260 0.Nkl....IEND.B`
00000130: 82
and then put that through my filter and display it again:
xxd -p image.png | sed -E 's/(..)../\1/g' | xxd -r -pp | xxd
00000000: 894e 0d1a 0000 4944 0000 0000 0200 008c .N....ID........
00000010: c700 0441 4100 8ffc 0500 2048 4d00 2600 ...AA..... HM.&.
00000020: 8400 0000 e800 3000 6000 9800 70ba 3c00 ......0.`...p.<.
00000030: 094c 4500 ff00 ff67 6400 0062 4702 0b64 .LE....gd..bG..d
00000040: 0007 4945 e41c 0850 9200 0049 4148 6318 ..IE...P...IAHc.
00000050: a30c 4320 5454 5454 5454 8f28 3100 0f7f ..C TTTTTT.(1...
00000060: afdd 0025 4574 6165 6365 7400 3030 302d ...%Etaecet.000-
00000070: 3831 3a38 332b 3030 d5d3 0000 7458 6474 81:83+00....tXdt
00000080: 3a6f 6979 3232 2d34 3254 3030 3a32 303a :oiy22-42T00:20:
00000090: 304e 6c00 0045 4442 82 0Nl..EDB.
Or in Perl
#!/usr/bin/perl
use strict;
use warnings;
open my $ifh, '<', $ARGV[0] or die "Need an input filename";
binmode $ifh;
open my $ofh, '>', $ARGV[1] or die "Need an output filename";
binmode $ofh;
my $x;
while (read $ifh,$x,2) {
my $num = pack 'C',ord($x); # or ord(substr $x,1) for odd bytes
print $ofh $num;
}
I am not sure if there such shell/bash built in command. seems like you need to write you own bash script for doing so.

Why TCP FIN/ACK packets have payload data but length is 0?

I use tcpdump to capture some data, but found that the FIN and ACK packets have some payload while the length is 0. Can anyone explain that? When I use Wireshark to see the pcap file, all is right. Why is that?
20:56:05.174314 IP 10.0.2.15.20281 > 192.168.4.80.21224: Flags [.], ack 1721, win 33232, length 0
0x0000: 0004 0001 0006 0800 2793 4e00 0000 0800 ........'.N.....
0x0010: 4500 0028 417a 4000 4006 284f 0a00 020f E..(Az#.#.(O....
0x0020: c0a8 0450 4f39 52e8 b35c bf82 f4d8 b0ba ...PO9R..\......
0x0030: 5010 81d0 d121 0000 4745 5420 2f75 7365 P....!..GET./use
0x0040: 7273 2f31 3f75 7365 rs/1?use
20:56:15.179096 IP 10.0.2.15.20281 > 192.168.4.80.21224: Flags [F.], seq 649, ack 1721, win 33232, length 0
0x0000: 0004 0001 0006 0800 2793 4e00 0000 0800 ........'.N.....
0x0010: 4500 0028 417b 4000 4006 284e 0a00 020f E..(A{#.#.(N....
0x0020: c0a8 0450 4f39 52e8 b35c bf82 f4d8 b0ba ...PO9R..\......
0x0030: 5011 81d0 d121 0000 0000 0000 0000 2e31 P....!.........1
0x0040: 2035 3030 2049 6e74 .500.Int
20:56:15.179528 IP 192.168.4.80.21224 > 10.0.2.15.20281: Flags [.], ack 650, win 65535, length 0
0x0000: 0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
0x0010: 4500 0028 73f3 0000 4006 35d6 c0a8 0450 E..(s...#.5....P
0x0020: 0a00 020f 52e8 4f39 f4d8 b0ba b35c bf83 ....R.O9.....\..
0x0030: 5010 ffff 2438 0000 0000 0000 0000 2e31 P...$8.........1
0x0040: 2035 3030 2049 6e74 6572 6e61 6c20 .500.Internal.
20:56:15.181826 IP 192.168.4.80.21224 > 10.0.2.15.20281: Flags [F.], seq 1721, ack 650, win 65535, length 0
0x0000: 0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
0x0010: 4500 0028 73f5 0000 4006 35d4 c0a8 0450 E..(s...#.5....P
0x0020: 0a00 020f 52e8 4f39 f4d8 b0ba b35c bf83 ....R.O9.....\..
0x0030: 5011 ffff 2437 0000 0000 0000 0000 7365 P...$7........se
0x0040: 7273 2f31 3f75 7365 725f 6964 3d35 rs/1?user_id=5
20:56:15.181884 IP 10.0.2.15.20281 > 192.168.4.80.21224: Flags [.], ack 1722, win 33232, length 0
0x0000: 0004 0001 0006 0800 2793 4e00 0000 0800 ........'.N.....
0x0010: 4500 0028 eaf7 4000 4006 7ed1 0a00 020f E..(..#.#.~.....
0x0020: c0a8 0450 4f39 52e8 b35c bf83 f4d8 b0bb ...PO9R..\......
0x0030: 5010 81d0 a266 0000 4745 5420 2f75 7365 P....f..GET./use
0x0040: 7273 2f31 3f75 7365 rs/1?use

Get a substring from a file shell script [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
i need get a substring from a file shell script
i need a little help with this shell script. I have a variable, represents a IP/TCP header. I need filter a traffic capture by the header selected.
> var=ttl 128 (only TTL=128)
>
> tcpdump -Xvv -n -i eth0 -c 300 > capture.txt 2>/dev/null
>
I'm trying using grep command, but only have the line with ttl 128, don't the ip source and destination
> grep -i "$var" capture.txt > resultGrep.txt
The result of the tcpdump command is some like this
15:29:18.164566 IP (tos 0x0, ttl 1, id 2394, offset 0, flags [none], proto UDP (17), length 125)
10.0.0.155.58363 > 239.255.255.254.1900: UDP, length 97
0x0000: 4600 0024 0000 0000 0102 3ad3 0a00 0000 F..$......:.....
0x0010: e000 0001 9404 0000 1101 ebfe 0000 0000 ................
0x0020: 0300 0000 0000 0000 0000 0000 0000 ..............
15:29:18.164566 IP (tos 0x0, ttl 128, id 2394, offset 0, flags [none], proto UDP (17), length 125)
10.0.0.131.58363 > 239.255.255.250.1900: UDP, length 97
0x0000: 4600 0024 0000 0000 0102 3ad3 0a00 0000 F..$......:.....
0x0010: e000 0001 9404 0000 1101 ebfe 0000 0000 ................
0x0020: 0300 0000 0000 0000 0000 0000 0000 ..............
15:29:18.164566 IP (tos 0x0, ttl 64, id 2394, offset 0, flags [none], proto UDP (17), length 125)
10.0.0.147.58363 > 239.255.255.255.1900: UDP, length 97
0x0000: 4600 0024 0000 0000 0102 3ad3 0a00 0000 F..$......:.....
0x0010: e000 0001 9404 0000 1101 ebfe 0000 0000 ................
0x0020: 0300 0000 0000 0000 0000 0000 0000 ..............
I need have ip address source and ip address destination from packets with ttl 128 header, in the example the output result must be
10.0.0.131.58363 > 239.255.255.250.1900
If your grep supports displaying a context -A, you can try
grep -A 1 -e 'ttl 128' capture.txt | grep '^ ' | cut -d: -f1
The first grep shows all lines with ttl 128 plus one following line. The second grep filters the lines starting with blanks. The final cut selects everything before the first :.

Checksum/CRC calculation

I am decoding a message sent by an wireless outdoor weather station. I have figured out what most of the protocol means except haven't been able to determine how to calculate the checksum. Here is what I've found so far:
0100 0111 0110 0000 0111 0100 0011 0010 1110 0110
|----|---------|--------------|---------|---------|
start id temp humidity checksum?
So in the above example the temperature was 11.6 degrees and 50% humidity. The "id" field changes to a "random" value when I replace the batteries in the outdoor transmitter. The last 8 bits only change when the id/temp/humidity changes so I'm fairly certain those are the checksum/crc bits.
Here are a bunch of different transmissions:
0100 1110 0101 0000 1001 0010 0011 0011 1010 0001
0100 1110 0101 0000 1001 0010 0011 0100 0011 0110
0100 0111 0110 0000 1001 0010 0011 0100 0110 1100
0100 0111 0110 0000 1001 0010 0011 0101 0101 1101
0100 0111 0110 0000 1001 0100 0011 0101 0000 0111
0100 0111 0110 0000 1001 0101 0011 0110 1010 0000
0100 0111 0110 0000 1001 1000 0011 0111 1101 0001
0100 0111 0110 0000 1010 1000 0011 0111 0110 0011
0100 0111 0110 0000 1010 1001 0011 0111 1001 0111
0100 0111 0110 0000 1010 1010 0011 0111 1011 1010
0100 0111 0110 0000 0111 0100 0011 0010 1110 0110
0100 0111 0110 0000 0111 0101 0011 0010 0001 0010
0100 0111 0110 0000 0111 0110 0011 0010 0011 1111
I'm not familiar with checksum/crc techniques, can anyone see what might be being used here or point me in a direction for how to determine this?
Seriously, your best bet is probably to go to the weather station and look at the manufacturer and model number, then either plug that into Google or send email to the manufacturer asking how the checksum is calculated.
If they baulk at releasing the information, tell them you're evaluating a solution and you may make a decision to buy ten to a hundred of their devices based on how easy it is to write software for it (information requests aren't delivered under oath and social engineering is still usually the easiest way to get at information you wouldn't normally get to see).
Hacked OK. Here's the C# code to generate the 5-th byte out of the previous 4 bytes.
/// <summary>A variant of CRC-8</summary>
/// <param name="data">Pass here the first 4 bytes of data, e.g. { 0x4E 0x50 0x92 0x33 }</param>
/// <returns>The computed SRC value, e.g. 0xA1 for the data specified above.</returns>
static byte src8( byte[] data )
{
byte res = 0;
foreach( byte b in data )
{
byte val = b;
for( int i = 0; i < 8; i++ )
{
byte tmp = (byte)( ( res ^ val ) & 0x80 );
res <<= 1;
if( 0 != tmp )
res ^= 0x31;
val <<= 1;
}
}
return res;
}

Resources