hexdump and xxd produce different outputs on red hat and ubuntu/mint - linux
Firstly , I am mentioning whatever happened on my linux mint/ubuntu system ...
udit#udit-Dabba ~/ah $ xxd -r -p input.txt output.txt
udit#udit-Dabba ~/ah $ cat input.txt
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29
udit#udit-Dabba ~/ah $ cat output.txt
` z2P��~�)
udit#udit-Dabba ~/ah $ hexdump -C output.txt
00000000 60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 |`...... ........|
00000010 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 02 00 15 00 15 7a 32 b1 08 |............z2..|
00000030 00 00 00 00 50 02 ff ff 7e 95 00 00 00 00 02 29 |....P...~......)|
00000040
udit#udit-Dabba ~/ah $ od -x output.txt
0000000 0060 0000 1400 2006 0000 0000 0000 0000
0000020 0000 0000 0000 0100 0000 0000 0000 0000
0000040 0000 0000 0000 0200 1500 1500 327a 08b1
0000060 0000 0000 0250 ffff 957e 0000 0000 2902
0000100
Now when I tried same thing on red hat , some strange behaviour is occuring.
[root#cc]$ xxd -r -p input.txt output.txt
[root#cc]$ cat input.txt
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29
[root#cc]$ cat output.txt
` P~)[root#cc]$ hexdump -C output.txt
00000000 60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 |`...... ........|
00000010 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 50 02 ff ff 7e 95 00 00 00 00 02 29 |....P...~......)|
00000030
[root#cc]$ od -x output.txt
0000000 0060 0000 1400 2006 0000 0000 0000 0000
0000020 0000 0000 0000 0100 0000 0000 0000 0000
0000040 0000 0000 0250 ffff 957e 0000 0000 2902
0000060
I am not able to find out why this discrepency occcuring between outputs of red hat and linux mint/ubuntu.
If possible tell me some awk/shell to perform the same in place of xxd -r -p
Pleae help me !!
EDIT:
On linux mint/ubuntu system ..
udit#udit-Dabba ~/ah $ hexdump -C input.txt
00000000 36 30 20 30 30 20 30 30 20 30 30 20 30 30 20 31 |60 00 00 00 00 1|
00000010 34 20 30 36 20 32 30 20 30 30 20 30 30 20 30 30 |4 06 20 00 00 00|
00000020 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 | 00 00 00 00 00 |
00000030 0a 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 |.00 00 00 00 00 |
00000040 30 30 20 30 30 20 30 31 20 30 30 20 30 30 20 30 |00 00 01 00 00 0|
00000050 30 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 |0 00 00 00 00 00|
00000060 20 0a 30 30 20 30 30 20 30 30 20 30 30 20 30 30 | .00 00 00 00 00|
00000070 20 30 30 20 30 30 20 30 32 20 30 30 20 31 35 20 | 00 00 02 00 15 |
00000080 30 30 20 31 35 20 37 41 20 33 32 20 42 31 20 30 |00 15 7A 32 B1 0|
00000090 38 20 0a 30 30 20 30 30 20 30 30 20 30 30 20 35 |8 .00 00 00 00 5|
000000a0 30 20 30 32 20 46 46 20 46 46 20 37 45 20 39 35 |0 02 FF FF 7E 95|
000000b0 20 30 30 20 30 30 20 30 30 20 30 30 20 30 32 20 | 00 00 00 00 02 |
000000c0 32 39 20 0a 0a 0a 0a 0a |29 .....|
000000c8
On red hat machine ..
[root#cc]$ hexdump -C input.txt
00000000 36 30 20 30 30 20 30 30 20 30 30 20 30 30 20 31 |60 00 00 00 00 1|
00000010 34 20 30 36 20 32 30 20 30 30 20 30 30 20 30 30 |4 06 20 00 00 00|
00000020 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 0a | 00 00 00 00 00.|
00000030 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 30 |00 00 00 00 00 0|
00000040 30 20 30 30 20 30 31 20 30 30 20 30 30 20 30 30 |0 00 01 00 00 00|
00000050 20 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 | 00 00 00 00 00 |
00000060 0a 30 30 20 30 30 20 30 30 20 30 30 20 30 30 20 |.00 00 00 00 00 |
00000070 30 30 20 30 30 20 30 32 20 30 30 20 31 35 20 30 |00 00 02 00 15 0|
00000080 30 20 31 35 20 37 41 20 33 32 20 42 31 20 30 38 |0 15 7A 32 B1 08|
00000090 20 0a 30 30 20 30 30 20 30 30 20 30 30 20 35 30 | .00 00 00 00 50|
000000a0 20 30 32 20 46 46 20 46 46 20 37 45 20 39 35 20 | 02 FF FF 7E 95 |
000000b0 30 30 20 30 30 20 30 30 20 30 30 20 30 32 20 32 |00 00 00 00 02 2|
000000c0 39 20 20 0a |9 .|
000000c4
I have no idea whats happening, but the third line is being eaten on red hat.
This is what you asked for. Reverse hexdump with awk+bash.
echo 'ibase=16' | cat - hdump | tr ' ' \; | bc | awk '{printf("%c",$0)}'
replace hdump with input.txt
echo 'ibase=16' | cat - input.txt | tr ' ' \; | bc | awk '{printf("%c",$0)}' > output.txt
codes were found here:
http://www.unix.com/shell-programming-scripting/132294-reverse-hexdump-without-xxd.html#post302404631
A late answer, but hopefully may help to somebody else reading this:
xxd had buggy handling of spaces in "-r -p" mode. As it scanned the characters, if it hit 2 consecutive non-hex characters, it would throw away everything until the next newline or end of file. If the second of the 2 consecutive characters was itself the newline character (i.e., the line ended with a space followed by a newline), the net effect would be to discard the entire next line.
I am pretty certain this is what caused Udit's problem. (I think the hexdump he later added of input.txt did not actually reflect the original state of that file.)
It looks like xxd is primarily maintained as part of the vim distribution. I looked at the xxd source included with vim 7.3, and it has the buggy handling problem fixed. Unfortunately nobody bothers to update the version #, so it still reports the same xxd 1.10 version string.
Try creating this test file (with 1 blank space at the end of the first line and none on the second; should be a 22 byte file on Linux/Unix; if you run "xxd -p" on it, it should report 3230200a34652034662035340a34322034312034340a):
20
4e 4f 54
42 41 44
If you run "xxd -r -p" on this, with a version of xxd exhibiting the buggy handling, you should see it print BAD. With a newer (fixed) version of xxd, it'll print NOTBAD.
If you run into this buggy handling and updating to a newer version of xxd is not an option, you can avoid this problem with some sed preprocessing, e.g.,
$ sed -e 's/ *//g' -e '/^$/d' input.txt | xxd -r -p > output.txt
I'd say that you have some invisible character in the input.txt file that makes the third line look invalid somehow.
And maybe the invalid character is encoding-dependent, and that's why it works in one machine and not in other.
It would be helpful to see the result of hexdump -C input.txt to check that.
$ cat foo
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29
$ cat foo | xxd -r -p | hexdump -C
00000000 60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 |`...... ........|
00000010 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 02 00 15 00 15 7a 32 b1 08 |............z2..|
00000030 00 00 00 00 50 02 ff ff 7e 95 00 00 00 00 02 29 |....P...~......)|
00000040
Related
.txt file filter for a specific HEX snipet
I have a .txt that i want to import into excel to plot the values converted from hex. The textfile is much larger (up to 40000 lines) so doing it manually is not an option. here is an example of the .txt file: Line DCE(hex) 1 2 3 4 41 14 00 00 9a 7e 00 00 5 00 00 00 00 00 00 00 00 6 00 01 6d 45 7 8 9 10 41 14 00 00 9a 99 00 00 11 00 00 00 00 00 00 00 00 12 00 01 88 45 13 14 15 16 41 14 00 00 9a b0 00 00 17 00 00 00 00 00 00 00 00 18 00 01 9f 45 19 20 21 22 41 14 00 00 9a c7 00 00 23 00 00 00 00 00 00 00 00 24 00 01 b6 45 25 26 27 28 41 14 00 00 9a df 00 00 29 00 00 00 00 00 00 00 00 30 00 01 ce 45 from the DCE(hex) line i need only the part i have written in quotation marks here: 4 41 14 00 00 "9a 7e" 00 00 5 00 00 00 00 00 00 00 00 6 00 01 6d 45 7 8 9 10 41 14 00 00 "9a 99" 00 00 11 00 00 00 00 00 00 00 00 12 00 01 88 45 13 14 15 16 41 14 00 00 "9a b0" 00 00 17 00 00 00 00 00 00 00 00 18 00 01 9f 45 i want the values at the end like this so i can import it in to excel and plot a diagramm. How can i do this the best way? 1 9a 7e 2 9a 99 3 9a b0 4 9a c7 Thank you very much!
Socat corrupt image file
I use 'socat TCP4-LISTEN:8080,fork EXEC:./bashttpd' for http server. when try to receive image file from client socat remove some byte and corrupt my image. correct: 01b0 0a 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 ..PNG........IHD 01c0 52 00 00 07 80 00 00 04 38 08 02 00 00 00 67 b1 R.......8.....g. 01d0 56 14 00 00 00 09 70 48 59 73 00 00 11 b0 00 00 V. incorrect:(socat -> read line -> xxh) 00000000: 89 50 4e 47 0d 0a .PNG.. 00000000: 1a 0a .. 00000000: 0d 49 48 44 52 07 80 04 38 08 02 67 b1 56 14 09 .IHDR...8..g.V how to solve this problem? thanks
Vim: calling xxd with system command in substitution results in conversion error
Background is that I have a log file that contains hex dumps that I want to convert with xxd to get that nice ASCII column that shows possible strings in the binary data.
The log file format looks like this:
My interesting hex dump:
00 53 00 6f 00 6d 00 65 00 20 00 74 00 65 00 78
00 74 00 20 00 65 00 78 00 61 00 6d 00 70 00 6c
00 65 00 20 00 75 00 73 00 69 00 6e 00 67 00 20
00 55 00 54 00 46 00 2d 00 31 00 36 00 20 00 69
00 6e 00 20 00 6f 00 72 00 64 00 65 00 72 00 20
00 74 00 6f 00 20 00 67 00 65 00 74 00 20 00 30
00 78 00 30 00 30 00 20 00 62 00 79 00 74 00 65
00 73 00 2e
Visually selecting the hex dump and do xxd -r -p followed by a xxd -g1 on the result does exactly what I'm aiming for.
However, since the number of dumps I want to convert are quite a few I would rather automate the process.
So I'm using the following substitute command to do the conversion:
:%s/\(\x\{2\} \?\)\{16\}\_.*/\=system('xxd -g1',system('xxd -r -p',submatch(0)))
The expression matches the entire hex dump in the log file. The match is sent to xxd -r -p as stdin and its output is used as stdin for xxd -g1.
Well, that's the idea at least.
The thing is that the above almost works. It produces the following result:
My interesting hex dump:
00000000: 01 53 01 6f 01 6d 01 65 01 20 01 74 01 65 01 78 .S.o.m.e. .t.e.x
00000010: 01 74 01 20 01 65 01 78 01 61 01 6d 01 70 01 6c .t. .e.x.a.m.p.l
00000020: 01 65 01 20 01 75 01 73 01 69 01 6e 01 67 01 20 .e. .u.s.i.n.g.
00000030: 01 55 01 54 01 46 01 2d 01 31 01 36 01 20 01 69 .U.T.F.-.1.6. .i
00000040: 01 6e 01 20 01 6f 01 72 01 64 01 65 01 72 01 20 .n. .o.r.d.e.r.
00000050: 01 74 01 6f 01 20 01 67 01 65 01 74 01 20 01 30 .t.o. .g.e.t. .0
00000060: 01 78 01 30 01 30 01 20 01 62 01 79 01 74 01 65 .x.0.0. .b.y.t.e
00000070: 01 73 01 2e .s..
All 00 bytes have mysteriously transformed into 01.
It should have produced the following:
My interesting hex dump:
00000000: 00 53 00 6f 00 6d 00 65 00 20 00 74 00 65 00 78 .S.o.m.e. .t.e.x
00000010: 00 74 00 20 00 65 00 78 00 61 00 6d 00 70 00 6c .t. .e.x.a.m.p.l
00000020: 00 65 00 20 00 75 00 73 00 69 00 6e 00 67 00 20 .e. .u.s.i.n.g.
00000030: 00 55 00 54 00 46 00 2d 00 31 00 36 00 20 00 69 .U.T.F.-.1.6. .i
00000040: 00 6e 00 20 00 6f 00 72 00 64 00 65 00 72 00 20 .n. .o.r.d.e.r.
00000050: 00 74 00 6f 00 20 00 67 00 65 00 74 00 20 00 30 .t.o. .g.e.t. .0
00000060: 00 78 00 30 00 30 00 20 00 62 00 79 00 74 00 65 .x.0.0. .b.y.t.e
00000070: 00 73 00 2e .s..
What am I not getting here?
Of course I can use macros and other ways of doing this, but I want to understand why my substitution command doesn't do what I expect.
Edit:
For anyone that want to achieve the same thing I provide the substitution expression that works on an entire file. The expression above was only for testing purposes using the log file example also from above.
The one below is the one that performs a correct conversion, modified based on the information Kent provided in his answer.
:%s/\(\(\x\{2\} \)\{16\}\_.\)\+/\=system('xxd -p -r | xxd -g1',submatch(0))
very likely, the problem is string conversion in the system() The input will be converted into a string by vim, so does the output of your first xxd command.
You can try to extract that hex parts into a file. then:
xxd -r -p theFile|vim -
And then calling the system('xxd -g1', alltext), you are gonna get something else than 00 too.
This doesn't work in the same way of a pipe (xxd ...|xxd...). But unfortunately, the system() function doesn't accept pipes.
If you want to fix your :s command, you need to call systemlist() on your first xxd call to get the data in binary format, then pass it to the 2nd xxd:
:%s/\(\x\{2\} \?\)\{16\}\_.*/\=system('xxd -g1',systemlist('xxd -r -p',submatch(0)))
The cmd above will generate the 00s. since there is no string conversion.
However, when working with some data format other than plain string, perhaps we can use filters instead of calling system(). It would be a lot eaiser. For your example:
2,$!xxd -r -p|xxd -g1
Difference In hexdump output on OSX and Linux
I am taking hex dump of the same file from two Machines ( Mac & Linux), but the order bytes looks different in both. I am something wrong? Hexdump output from OSX ➜ samples hexdump 1500_77807e3eb2eeacd9ac870c24103f5b_fno.bin 0000000 02 20 00 01 01 0b 1a 02 59 5a 2a 2a 31 32 33 00 0000010 40 00 05 4a fa a4 77 00 00 1c 28 40 28 02 17 ef 0000020 6a 36 00 e0 4c 01 b8 f5 32 20 a0 00 07 01 d6 00 0000030 02 00 00 c3 1f 00 01 40 1f 18 67 8f 00 2f 26 43 0000040 2d 40 96 30 89 40 11 4b a0 ff 28 34 00 35 00 02 0000050 02 58 40 bd 25 40 ec 40 01 01 40 a6 26 20 61 34 0000060 40 2d c2 5f 16 00 03 27 00 2d 11 be 00 4b 5d 0c 0000070 c0 bd 2c 40 2d 4d a0 be 06 72 40 2f 52 00 05 7e 0000080 00 96 5d 57 a0 5c 80 2d 5c be 04 b0 5d 61 bc 03 0000090 00 00 41 1c f7 a0 80 1f 05 c5 70 7d 40 c2 ce 2a 00000a0 e9 00 0e 2c dd 00 0e 24 ef 00 0e b7 f0 83 54 61 00000b0 d7 03 bb 3c 43 57 03 c6 c7 be 99 84 60 44 36 03 00000c0 54 41 d7 03 b0 e2 a2 96 00 e1 00 2e af 5c bc 43 00000d0 b7 03 ad 6d c0 5c 41 4e 9f 6c 28 00 2e 9c c9 28 00000e0 00 2e b9 43 dc 41 1d b9 c2 c4 43 b7 03 ba 51 27 00000f0 03 e7 03 bb af a0 2e 0e a6 5d dc bc 43 17 d0 a0 0000100 40 80 1f d3 30 80 c3 54 62 cc 62 ac c0 0c 11 00 0000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * 00005d0 00 00 00 00 00 00 00 00 00 00 00 00 00005dc Hexdump output of the same file from Ubuntu server SSH ed in from OSX exdump 1500_77807e3eb2eeacd9ac870c24103f5b_fno.bin 0000000 2002 0100 0b01 021a 5a59 2a2a 3231 0033 0000010 0040 4a05 a4fa 0077 1c00 4028 0228 ef17 0000020 366a e000 014c f5b8 2032 00a0 0107 00d6 0000030 0002 c300 001f 4001 181f 8f67 2f00 4326 0000040 402d 3096 4089 4b11 ffa0 3428 3500 0200 0000050 5802 bd40 4025 40ec 0101 a640 2026 3461 0000060 2d40 5fc2 0016 2703 2d00 be11 4b00 0c5d 0000070 bdc0 402c 4d2d bea0 7206 2f40 0052 7e05 0000080 9600 575d 5ca0 2d80 be5c b004 615d 03bc 0000090 0000 1c41 a0f7 1f80 c505 7d70 c240 2ace 00000a0 00e9 2c0e 00dd 240e 00ef b70e 83f0 6154 00000b0 03d7 3cbb 5743 c603 bec7 8499 4460 0336 00000c0 4154 03d7 e2b0 96a2 e100 2e00 5caf 43bc 00000d0 03b7 6dad 5cc0 4e41 6c9f 0028 9c2e 28c9 00000e0 2e00 43b9 41dc b91d c4c2 b743 ba03 2751 00000f0 e703 bb03 a0af 0e2e 5da6 bcdc 1743 a0d0 0000100 8040 d31f 8030 54c3 cc62 ac62 0cc0 0011 0000110 0000 0000 0000 0000 0000 0000 0000 0000 * Even the 1st word is different 0220 & 2002
It looks like you're having an issue with both endianness and hexdump defaulting to single bytes on your OSX environment and 2-byte words on your Linux environment. To get identical output you might need to explicitly provide a format string. Running hexdump -e \"%07_ax\ \"\ 16/1\ \"\ %02x\"\ \"\\n\" 1500_77807e3eb2eeacd9ac870c24103f5b_fno.bin should give you the same result in both environments. The format string is "%08_ax " 16/1 " %02x" "\n" with escaping.
What is wrong with my simple function-call intel x64 machine code?
Here is my function calling code: (Instructions start at 0x00403000) 6A 28 (Stack Push 0x28) EB 0A (Jump 10) 40 80 04 25 03 10 40 00 05 (Add 5 to address 0x00401003) C3 (Near return) FF 14 25 04 30 40 00 (Call 0x00403004) This causes my program to blow up instead of calling MessageBoxA later in the code. Here is the functional code that works but without the function call: (Instructions start at 0x00403000) 6A 28 (Stack Push 0x28) EB 0A (Jump 10) 40 80 04 25 03 10 40 00 05 (Add 5 to address 0x00401003) C3 (Near return) 40 80 04 25 03 10 40 00 05 (Add 5 to address 0x00401003) Here's the code after these snippets: 48 B9 00 00 00 00 00 00 00 00 (Copy 0 to 64-register 1) 48 BA 00 10 40 00 00 00 00 00 (Copy 0x00401000 to 64-register 2) 49 B8 06 10 40 00 00 00 00 00 (Copy 0x00401006 to 64-register 3) 49 B9 00 00 00 00 00 00 00 00 (Copy 0 to 64-register 4) FF 14 25 78 20 40 00 (Call MessageBoxA) 48 B9 00 00 00 00 00 00 00 00 (Copy 0 to 64-register 1) FF 14 25 88 20 40 00 (Call End Process)