As the number of fragments in a file increases, you can watch the number of bytes used in the MFT record grow. As expected, it grows in increments of 8 until it fills every last byte of the 1024 byte base record, before splitting into a second record. At first blush, you might expect the answer to be "1024 - headerlength."
However, as that second record grows, it doesn't use every byte of that record. Instead, it tops out at 1016, leaving 8 bytes of the record unused when it splits into a 3rd record (see below). If I continue splitting the runs, the third record will also split, again at 1016 instead of 1024.
NTFS is usually highly efficient in handling MFT records, which makes this behavior highly suspect. It smells like a bug, although I suppose there could be some obscure reason it's necessary.
But then I started experimenting with 4k MFT records (format /L). And both the base records and the extension records always use the full 4096 bytes.
So now it really looks like a bug. Although I'm not sure if the bug is that 1k records top out at 1016, or that 4k records don't top out at 4088. Either there's a good reason to reserve those final bytes or there isn't, right?
In addition to wasting a precious resource, it makes computing 'available' space on MFT records really complicated. Is the formula really "assume records will grow to the MFT record size, unless it's an extension record, then subtract 8, unless it's a 4k record, then don't do the subtraction?" Or does this behavior imply there's some other criteria here? Is there a trick?
FWIW, here's the 1016 byte record right before the split (as seen in a disk editor). You can see the end of attributes marker (FFFFFFFF 82794711) followed by 8 empty bytes. The trailing 02 00 is the Updatebytes used by MFT records. After the next split, the size of this record drops in half.
Offset | 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F | ASCII
------------------------------------------------------------------------------------------------
00C8788400 | 46 49 4C 45 30 00 03 00 E2 2F 31 C5 11 00 00 00 | FILE0..../1.....
00C8788410 | 2F 00 00 00 38 00 01 00 F8 03 00 00 00 04 00 00 | /...8...........
00C8788420 | 51 31 00 00 00 00 8F 00 01 00 00 00 21 1E 02 00 | Q1..........!...
00C8788430 | 02 00 11 01 00 00 00 00 80 00 00 00 B8 03 00 00 | ................
00C8788440 | 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00C8788450 | 3E 08 00 00 00 00 00 00 40 00 00 00 00 00 00 00 | >.......#.......
00C8788460 | 00 F0 83 00 00 00 00 00 62 EB 83 00 00 00 00 00 | ........b.......
00C8788470 | 62 EB 83 00 00 00 00 00 41 01 80 F0 9E 03 11 01 | b.......A.......
00C8788480 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788490 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C87884A0 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C87884B0 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C87884C0 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C87884D0 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C87884E0 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C87884F0 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788500 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C8788510 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788520 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788530 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C8788540 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788550 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788560 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C8788570 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788580 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788590 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C87885A0 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C87885B0 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C87885C0 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C87885D0 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C87885E0 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C87885F0 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 02 00 | ................
00C8788600 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788610 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788620 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C8788630 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788640 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788650 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C8788660 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788670 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788680 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C8788690 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C87886A0 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C87886B0 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C87886C0 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C87886D0 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C87886E0 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C87886F0 | FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788700 | 11 01 FF 11 01 FF 31 01 32 A1 FE 11 01 FF 21 01 | ......1.2.....!.
00C8788710 | 3A FE 11 01 FF 21 01 C3 F8 11 01 FF 21 01 BD FB | :....!......!...
00C8788720 | 11 01 FF 11 01 FF 21 01 52 FD 11 01 FF 11 01 FF | ......!.R.......
00C8788730 | 11 01 FF 21 01 A5 A9 11 01 FF 21 01 A2 EF 11 01 | ...!......!.....
00C8788740 | FA 11 01 F9 11 01 FF 11 01 FF 11 01 FF 11 01 FD | ................
00C8788750 | 11 01 FF 11 01 FB 11 01 FF 11 01 F2 11 01 FB 11 | ................
00C8788760 | 01 FC 11 01 FD 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C8788770 | FD 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF | ................
00C8788780 | 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C8788790 | 01 FF 11 01 FF 11 01 FF 11 01 FF 21 01 F4 FE 21 | ...........!...!
00C87887A0 | 01 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 | ................
00C87887B0 | 01 FF 11 01 FF 11 01 FF 21 01 FD FE 11 01 FF 11 | ........!.......
00C87887C0 | 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 FF 11 01 | ................
00C87887D0 | FF 11 01 85 11 01 FF 11 01 FB 11 01 FF 11 01 FF | ................
00C87887E0 | 11 01 FF 11 01 FB 42 1F 07 3A 3D C5 01 00 00 00 | ......B..:=.....
00C87887F0 | FF FF FF FF 82 79 47 11 00 00 00 00 00 00 02 00 | .....yG.........
Related
I have a .txt that i want to import into excel to plot the values converted from hex. The textfile is much larger (up to 40000 lines) so doing it manually is not an option.
here is an example of the .txt file:
Line DCE(hex)
1
2
3
4 41 14 00 00 9a 7e 00 00
5 00 00 00 00 00 00 00 00
6 00 01 6d 45
7
8
9
10 41 14 00 00 9a 99 00 00
11 00 00 00 00 00 00 00 00
12 00 01 88 45
13
14
15
16 41 14 00 00 9a b0 00 00
17 00 00 00 00 00 00 00 00
18 00 01 9f 45
19
20
21
22 41 14 00 00 9a c7 00 00
23 00 00 00 00 00 00 00 00
24 00 01 b6 45
25
26
27
28 41 14 00 00 9a df 00 00
29 00 00 00 00 00 00 00 00
30 00 01 ce 45
from the DCE(hex) line i need only the part i have written in quotation marks here:
4 41 14 00 00 "9a 7e" 00 00
5 00 00 00 00 00 00 00 00
6 00 01 6d 45
7
8
9
10 41 14 00 00 "9a 99" 00 00
11 00 00 00 00 00 00 00 00
12 00 01 88 45
13
14
15
16 41 14 00 00 "9a b0" 00 00
17 00 00 00 00 00 00 00 00
18 00 01 9f 45
i want the values at the end like this so i can import it in to excel and plot a diagramm.
How can i do this the best way?
1 9a 7e
2 9a 99
3 9a b0
4 9a c7
Thank you very much!
I have this code:
from scapy.all import *
packet = rdpcap("hand.pcap")[0]
Now when I use print(packet) I get normal bytes
b"\x00\x00\x18\x00+\x00\x00\x00\x19\x81\xea\x02\x00\x00\x00\x00\x10\x00l\t\x00\x00\xdb\x00\x80\x00\x00\x00\xff\xff\xff\xff\xff\xff\x06\xd6\xaa\xb0x\xe9\x06\xd6\xaa\xb0x\xe9\x80V\x8cAn\xa0\x01\x00\x00\x00d\x00\x11\x15\x00\nHuawei281s\x01\x08\x82\x84\x8b\x96$0Hl\x03\x01\x01\x05\x04\x00\x02\x00\x00\x07\x06HR \x01\r\x14 \x01\x00#\x02\x13\x00*\x01\x022\x04\x0c\x12\x18`0\x14\x01\x00\x00\x0f\xac\x04\x01\x00\x00\x0f\xac\x04\x01\x00\x00\x0f\xac\x02\x0c\x00-\x1a\xad\x01\x1b\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00=\x16\x01\x00\x17\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7f\x08\x04\x00\x08\x00\x00\x00\x00#\xdd\x05\x00\x162\x80\x00\xdd\x08\x00P\xf2\x11\x02\x00\x00\x00\xdd\x1e\x00\x90L\x04\x08\xbf\x0c\xb2y\x81\x0f\xfa\xff\x00\x00\xfa\xff\x00\x00\xc0\x05\x00\x01\x00\x00\x00\xc3\x02\x00\x02\xdd\t\x00\x10\x18\x02\x01\x00\x1c\x00\x00\xdd\x18\x00P\xf2\x02\x01\x01\x00\x00\x03\xa4\x00\x00'\xa4\x00\x00BC^\x00b2/\x00\xdd\x1d\x00\x90L\\\x02\x01\n\x00\x08\x07\x01\x0f\x00\x00\x00\x00\x00\x01\n\x01\x01\x01\x01\x0f\x00\x00\x00\x00\x00d\x08\x06~"
When I use hexdump(packet) I get this
0000 00 00 18 00 2B 00 00 00 19 81 EA 02 00 00 00 00 ....+...........
0010 10 00 6C 09 00 00 DB 00 80 00 00 00 FF FF FF FF ..l.............
0020 FF FF 06 D6 AA B0 78 E9 06 D6 AA B0 78 E9 80 56 ......x.....x..V
0030 8C 41 6E A0 01 00 00 00 64 00 11 15 00 0A 48 75 .An.....d.....Hu
0040 61 77 65 69 32 38 31 73 01 08 82 84 8B 96 24 30 awei281s......$0
0050 48 6C 03 01 01 05 04 00 02 00 00 07 06 48 52 20 Hl...........HR
0060 01 0D 14 20 01 00 23 02 13 00 2A 01 02 32 04 0C ... ..#...*..2..
0070 12 18 60 30 14 01 00 00 0F AC 04 01 00 00 0F AC ..`0............
0080 04 01 00 00 0F AC 02 0C 00 2D 1A AD 01 1B FF FF .........-......
0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00a0 00 00 00 00 00 3D 16 01 00 17 00 00 00 00 00 00 .....=..........
00b0 00 00 00 00 00 00 00 00 00 00 00 00 00 7F 08 04 ................
00c0 00 08 00 00 00 00 40 DD 05 00 16 32 80 00 DD 08 ......#....2....
00d0 00 50 F2 11 02 00 00 00 DD 1E 00 90 4C 04 08 BF .P..........L...
00e0 0C B2 79 81 0F FA FF 00 00 FA FF 00 00 C0 05 00 ..y.............
00f0 01 00 00 00 C3 02 00 02 DD 09 00 10 18 02 01 00 ................
0100 1C 00 00 DD 18 00 50 F2 02 01 01 00 00 03 A4 00 ......P.........
0110 00 27 A4 00 00 42 43 5E 00 62 32 2F 00 DD 1D 00 .'...BC^.b2/....
0120 90 4C 5C 02 01 0A 00 08 07 01 0F 00 00 00 00 00 .L\.............
0130 01 0A 01 01 01 01 0F 00 00 00 00 00 64 08 06 7E ............d..~
And in that is my SSID with name Huawei281s in scambled in
When I use print(packet.show)
I get this
<bound method Packet.show of <RadioTap version=0 pad=0 len=24 present=TSFT+Flags+Channel+dBm_AntSignal mac_timestamp=48922905 Flags=FCS ChannelFrequency=2412 ChannelFlags= dBm_AntSignal=-37dBm notdecoded='\x00' |<Dot11FCS subtype=8 type=Management proto=0 FCfield= ID=0 addr1=ff:ff:ff:ff:ff:ff addr2=06:d6:aa:b0:78:e9 addr3=06:d6:aa:b0:78:e9 SC=22144 fcs=0x7e060864 |<Dot11Beacon timestamp=6986547596 beacon_interval=100 cap=res8+short-slot+res12+ESS+privacy |<Dot11Elt ID=SSID len=10 info='Huawei281s' |<Dot11EltRates ID=1 len=8 rates=[0x82, 0x84, 0x8b, 0x96, 0x24, 0x30, 0x48, 0x6c] |<Dot11Elt ID=DSset len=1 info='\x01' |<Dot11Elt ID=TIM len=4 info='\x00\x02\x00\x00' |<Dot11EltCountry ID=7 len=6 country_string='HR ' descriptors=[<Dot11EltCountryConstraintTriplet first_channel_number=1 num_channels=13 mtp=20 |>] pad=32 |<Dot11EltRates ID=1 len=0 |<Dot11Elt ID=35 len=2 info='\x13\x00' |<Dot11Elt ID=ERPinfo len=1 info='\x02' |<Dot11Elt ID=ESRates len=4 info='\x0c\x12\x18`' |<Dot11EltRSN ID=48 len=20 version=1 group_cipher_suite=<RSNCipherSuite oui=0xfac cipher=CCMP |> nb_pairwise_cipher_suites=1 pairwise_cipher_suites=[<RSNCipherSuite oui=0xfac cipher=CCMP |>] nb_akm_suites=1 akm_suites=[<AKMSuite oui=0xfac suite=PSK |>] mfp_capable=0 mfp_required=0 gtksa_replay_counter=0 ptksa_replay_counter=3 no_pairwise=0 pre_auth=0 reserved=0 |<Dot11Elt ID=HTCapabilities len=26 info='\xad\x01\x1b\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' |<Dot11Elt ID=HTinfo len=22 info='\x01\x00\x17\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' |<Dot11Elt ID=ExtendendCapatibilities len=8 info='\x04\x00\x08\x00\x00\x00\x00#' |<Dot11EltVendorSpecific ID=221 len=5 oui=0x1632 info='\x80\x00' |<Dot11EltVendorSpecific ID=221 len=8 oui=0x50f2 info='\x11\x02\x00\x00\x00' |<Dot11EltVendorSpecific ID=221 len=30 oui=0x904c info='\x04\x08\xbf\x0c\xb2y\x81\x0f\xfa\xff\x00\x00\xfa\xff\x00\x00\xc0\x05\x00\x01\x00\x00\x00\xc3\x02\x00\x02' |<Dot11EltVendorSpecific ID=221 len=9 oui=0x1018 info='\x02\x01\x00\x1c\x00\x00' |<Dot11EltVendorSpecific ID=221 len=24 oui=0x50f2 info="\x02\x01\x01\x00\x00\x03\xa4\x00\x00'\xa4\x00\x00BC^\x00b2/\x00" |<Dot11EltVendorSpecific ID=221 len=29 oui=0x904c info='\\\x02\x01\n\x00\x08\x07\x01\x0f\x00\x00\x00\x00\x00\x01\n\x01\x01\x01\x01\x0f\x00\x00\x00\x00\x00' |>>>>>>>>>>>>>>>>>>>>>>>
And there is my SSID under info
and for print(type(packet.show)) I get
<class 'method'>
And when I use packet.show() I get bunch of data with this
###[ 802.11 Information Element ]###
ID = SSID
len = 10
info = 'Huawei281s'
So im wondering how can I extract my SSID from packet
In this specific case you could use
packet[Dot11Beacon].network_stats()
But have a look at the doc for a more in depth tutorial. it seems you haven't really looked into Scapy's syntax :)
I use node-canvas;
....
var content = canvas.toBuffer();
var length=content.length;
console.log(content);
result:
<SlowBuffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 30 00 00 00 30 08 06 00 00 00 57 02 f9 87 00 00 00 06 62 4b 47 44 00 ff 00 ff 00 ff a0 bd a7 93 ...>
And
var buf=new SlowBuffer(length);
buf.write(content.toString());
console.log(buf);
result:
<Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 01 00 28 00 28 00 00 ff db 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 0a 0c 14 0d 0c 0b 0b 0c 19 12 13 0f 14 ...>
They are not equal:( ,so when I store it into redis ,I can't get it back;
Depending on the format you want, you can use the following methods:
buf.toJSON() // Straight to JSON format
buf.toString('utf8') ; // UTF8 format
Read on for more alternatives: https://nodejs.org/api/buffer.html#buffer_buf_tostring_encoding_start_end
You can't just call Buffer.toString and assume that everything will be all right, since the default encoding is utf8. If you want to encode binary data, you need base64 encoding.
Do you know of any good Buffer inspection library for Node.js?
It would be pretty useful to print out a Buffer along with its utf8 content.
Example:
Byte | HEX | UTF-8 | Name
-----+-----+-------+-----------
0 | 48 | H | UPPER H
1 | 65 | e | LOWER E
2 | 6c | l | LOWER L
3 | 6c | l | LOWER L
4 | 6f | o | LOWER O
5 | 20 | | WHITESPACE
6 | 57 | W | UPPER W
7 | 6f | o | LOWER O
8 | 72 | r | LOWER R
9 | 6c | l | LOWER L
10 | 64 | d | LOWER D
Yesterday I made a simple module called hex. It's like the hxd program. No config necessary, easy to use, for debugging purposes.
var hex = require("hex");
hex(buffer);
Output:
Offset 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
000000 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
000010 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
000020 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
000030 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
000040 54 41 47 42 72 65 61 6B 69 6E 67 20 54 68 65 20 TAGBreaking The
000050 4C 61 77 00 00 00 00 00 00 00 00 00 00 00 00 00 Law.............
000060 00 4A 75 64 61 73 20 50 72 69 65 73 74 00 00 00 .Judas Priest...
000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 42 ...............B
000080 72 69 74 69 73 68 20 53 74 65 65 6C 00 00 00 00 ritish Steel....
000090 00 00 00 00 00 00 00 00 00 00 00 00 00 31 39 38 .............198
0000A0 30 47 72 65 61 74 20 73 6F 6E 67 21 00 00 00 00 0Great song!....
0000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 33 89 ..............3.
Firstly , I am mentioning whatever happened on my linux mint/ubuntu system ...
udit#udit-Dabba ~/ah $ xxd -r -p input.txt output.txt
udit#udit-Dabba ~/ah $ cat input.txt
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29
udit#udit-Dabba ~/ah $ cat output.txt
` z2P��~�)
udit#udit-Dabba ~/ah $ hexdump -C output.txt
00000000 60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 |`...... ........|
00000010 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 02 00 15 00 15 7a 32 b1 08 |............z2..|
00000030 00 00 00 00 50 02 ff ff 7e 95 00 00 00 00 02 29 |....P...~......)|
00000040
udit#udit-Dabba ~/ah $ od -x output.txt
0000000 0060 0000 1400 2006 0000 0000 0000 0000
0000020 0000 0000 0000 0100 0000 0000 0000 0000
0000040 0000 0000 0000 0200 1500 1500 327a 08b1
0000060 0000 0000 0250 ffff 957e 0000 0000 2902
0000100
Now when I tried same thing on red hat , some strange behaviour is occuring.
[root#cc]$ xxd -r -p input.txt output.txt
[root#cc]$ cat input.txt
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29
[root#cc]$ cat output.txt
` P~)[root#cc]$ hexdump -C output.txt
00000000 60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 |`...... ........|
00000010 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 50 02 ff ff 7e 95 00 00 00 00 02 29 |....P...~......)|
00000030
[root#cc]$ od -x output.txt
0000000 0060 0000 1400 2006 0000 0000 0000 0000
0000020 0000 0000 0000 0100 0000 0000 0000 0000
0000040 0000 0000 0250 ffff 957e 0000 0000 2902
0000060
I am not able to find out why this discrepency occcuring between outputs of red hat and linux mint/ubuntu.
If possible tell me some awk/shell to perform the same in place of xxd -r -p
Pleae help me !!
EDIT:
On linux mint/ubuntu system ..
udit#udit-Dabba ~/ah $ hexdump -C input.txt
00000000 36 30 20 30 30 20 30 30 20 30 30 20 30 30 20 31 |60 00 00 00 00 1|
00000010 34 20 30 36 20 32 30 20 30 30 20 30 30 20 30 30 |4 06 20 00 00 00|
00000020 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 | 00 00 00 00 00 |
00000030 0a 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 |.00 00 00 00 00 |
00000040 30 30 20 30 30 20 30 31 20 30 30 20 30 30 20 30 |00 00 01 00 00 0|
00000050 30 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 |0 00 00 00 00 00|
00000060 20 0a 30 30 20 30 30 20 30 30 20 30 30 20 30 30 | .00 00 00 00 00|
00000070 20 30 30 20 30 30 20 30 32 20 30 30 20 31 35 20 | 00 00 02 00 15 |
00000080 30 30 20 31 35 20 37 41 20 33 32 20 42 31 20 30 |00 15 7A 32 B1 0|
00000090 38 20 0a 30 30 20 30 30 20 30 30 20 30 30 20 35 |8 .00 00 00 00 5|
000000a0 30 20 30 32 20 46 46 20 46 46 20 37 45 20 39 35 |0 02 FF FF 7E 95|
000000b0 20 30 30 20 30 30 20 30 30 20 30 30 20 30 32 20 | 00 00 00 00 02 |
000000c0 32 39 20 0a 0a 0a 0a 0a |29 .....|
000000c8
On red hat machine ..
[root#cc]$ hexdump -C input.txt
00000000 36 30 20 30 30 20 30 30 20 30 30 20 30 30 20 31 |60 00 00 00 00 1|
00000010 34 20 30 36 20 32 30 20 30 30 20 30 30 20 30 30 |4 06 20 00 00 00|
00000020 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 0a | 00 00 00 00 00.|
00000030 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 30 |00 00 00 00 00 0|
00000040 30 20 30 30 20 30 31 20 30 30 20 30 30 20 30 30 |0 00 01 00 00 00|
00000050 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 | 00 00 00 00 00 |
00000060 0a 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 |.00 00 00 00 00 |
00000070 30 30 20 30 30 20 30 32 20 30 30 20 31 35 20 30 |00 00 02 00 15 0|
00000080 30 20 31 35 20 37 41 20 33 32 20 42 31 20 30 38 |0 15 7A 32 B1 08|
00000090 20 0a 30 30 20 30 30 20 30 30 20 30 30 20 35 30 | .00 00 00 00 50|
000000a0 20 30 32 20 46 46 20 46 46 20 37 45 20 39 35 20 | 02 FF FF 7E 95 |
000000b0 30 30 20 30 30 20 30 30 20 30 30 20 30 32 20 32 |00 00 00 00 02 2|
000000c0 39 20 20 0a |9 .|
000000c4
I have no idea whats happening, but the third line is being eaten on red hat.
This is what you asked for. Reverse hexdump with awk+bash.
echo 'ibase=16' | cat - hdump | tr ' ' \; | bc | awk '{printf("%c",$0)}'
replace hdump with input.txt
echo 'ibase=16' | cat - input.txt | tr ' ' \; | bc | awk '{printf("%c",$0)}' > output.txt
codes were found here:
http://www.unix.com/shell-programming-scripting/132294-reverse-hexdump-without-xxd.html#post302404631
A late answer, but hopefully may help to somebody else reading this:
xxd had buggy handling of spaces in "-r -p" mode. As it scanned the characters, if it hit 2 consecutive non-hex characters, it would throw away everything until the next newline or end of file. If the second of the 2 consecutive characters was itself the newline character (i.e., the line ended with a space followed by a newline), the net effect would be to discard the entire next line.
I am pretty certain this is what caused Udit's problem. (I think the hexdump he later added of input.txt did not actually reflect the original state of that file.)
It looks like xxd is primarily maintained as part of the vim distribution. I looked at the xxd source included with vim 7.3, and it has the buggy handling problem fixed. Unfortunately nobody bothers to update the version #, so it still reports the same xxd 1.10 version string.
Try creating this test file (with 1 blank space at the end of the first line and none on the second; should be a 22 byte file on Linux/Unix; if you run "xxd -p" on it, it should report 3230200a34652034662035340a34322034312034340a):
20
4e 4f 54
42 41 44
If you run "xxd -r -p" on this, with a version of xxd exhibiting the buggy handling, you should see it print BAD. With a newer (fixed) version of xxd, it'll print NOTBAD.
If you run into this buggy handling and updating to a newer version of xxd is not an option, you can avoid this problem with some sed preprocessing, e.g.,
$ sed -e 's/ *//g' -e '/^$/d' input.txt | xxd -r -p > output.txt
I'd say that you have some invisible character in the input.txt file that makes the third line look invalid somehow.
And maybe the invalid character is encoding-dependent, and that's why it works in one machine and not in other.
It would be helpful to see the result of hexdump -C input.txt to check that.
$ cat foo
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29
$ cat foo | xxd -r -p | hexdump -C
00000000 60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 |`...... ........|
00000010 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 02 00 15 00 15 7a 32 b1 08 |............z2..|
00000030 00 00 00 00 50 02 ff ff 7e 95 00 00 00 00 02 29 |....P...~......)|
00000040