I have a hex string which I want to convert into a Wireshark pcap. Wireshark accepts a hex dump in the following format:
0000 00 00 00 00 00 aa 00 00 00 00 00 01 88 47 00 3e
0010 80 0a 00 00 d1 0a 10 00 89 02 20 01 05 46 00 00
0020 00 01 00 02 04 03 6d 64 31 02 03 6d 61 57 00 00
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0060 00 00 00 00 00
I have the following stream:
0000000000AA0000000000018847003E800A0000D10A100089022001054600000001000204036D643102036D615700000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
I have been experimenting with od -Ax -tc1 -v command in linux but can not seem to get the correct output. Does anyone know how this can be done?
First put the hex stream into a text file "a.txt",
0000000000AA0000000000018847003E800A0000D10A100089022001054600000001000204036D643102036D615700000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
then convert the text file into a binary file:
xxd -r -p a.txt a.bin
using od and text2pcap to convert the binary file to a pcap file.
od -Ax -tx1 -v a.bin | text2pcap - a.pcap
You can use the following script. This is just a workaround. od and hexdump will interpret 0 as the character '0', and use the byte 48, so you may not get proper output. The script reads two chars for 16 times, and offset is incremented accordingly. Pretty simple to understand I guess:)
#!/bin/bash
off=0
while [ 1 ]
do
printf "%04x " $off
for ((i=0;i<16;i++))
do
read -n 2 a
[ $? -ne 0 ] && echo && exit
echo -n "$a "
done
echo
off=`expr $off + 16`
done <test
where test is the file that holds the stream. I got the following output for your input.
0000 00 00 00 00 00 AA 00 00 00 00 00 01 88 47 00 3E
0010 80 0A 00 00 D1 0A 10 00 89 02 20 01 05 46 00 00
0020 00 01 00 02 04 03 6D 64 31 02 03 6D 61 57 00 00
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0060 00 00 00 00 00
Related
I'm analyzing this tiny ELF file:
00000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 |.ELF............|
00000010 02 00 3e 00 01 00 00 00 78 00 40 00 00 00 00 00 |..>.....x.#.....|
00000020 40 00 00 00 00 00 00 00 98 00 00 00 00 00 00 00 |#...............|
00000030 00 00 00 00 40 00 38 00 01 00 40 00 03 00 02 00 |....#.8...#.....|
00000040 01 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 40 00 00 00 00 00 00 00 40 00 00 00 00 00 |..#.......#.....|
00000060 7e 00 00 00 00 00 00 00 7e 00 00 00 00 00 00 00 |~.......~.......|
00000070 00 00 20 00 00 00 00 00 31 c0 ff c0 cd 80 00 2e |.. .....1.......|
00000080 73 68 73 74 72 74 61 62 00 2e 74 65 78 74 00 00 |shstrtab..text..|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000000d0 00 00 00 00 00 00 00 00 0b 00 00 00 01 00 00 00 |................|
000000e0 06 00 00 00 00 00 00 00 78 00 40 00 00 00 00 00 |........x.#.....|
000000f0 78 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 |x...............|
00000100 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000110 00 00 00 00 00 00 00 00 01 00 00 00 03 00 00 00 |................|
00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000130 7e 00 00 00 00 00 00 00 11 00 00 00 00 00 00 00 |~...............|
00000140 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000150 00 00 00 00 00 00 00 00 |........|
00000158
I found documentation on the ELF header and the program header and decoded both of those, but I'm having problems decoding what's after this (starting with 31 c0 ff c0 cd 80 00 2e). Judging by the "shstrtab" text, I am looking at the section table, but what does 31 c0 ff c0 cd 80 00 2e mean? Where is this part documented?
OK, judging by the information in the first 16 bytes of the header:
00000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 |.ELF............|
E L F | | '--- Pudding :) ---'
| '--- Little-endian (ELFDATA2LSB)
'------ 64-bit (ELFCLASS64)
we're dealing with a 64-bit ELF with little-endian encoding of multi-byte numbers. So the ELF header is the first 4 rows in the hex editor. We're interested in these fields in the last two rows of it:
Prog Hdr Tab offset Sect Hdr Tab offset
.----------^----------. .----------^----------.
00000020 40 00 00 00 00 00 00 00 98 00 00 00 00 00 00 00 |#...............|
00000030 00 00 00 00 40 00 38 00 01 00 40 00 03 00 02 00 |....#.8...#.....|
'-.-' '-.-' '-.-' '-.-' '-.-'
PHT entry size ---' | | | '-- Sect names in #2
PHT num entries ----------' | '-- SHT num entries
'-------- SHT entry size
So we know that the Program Headers Table starts at offset 0x40 in the file (right after this header) and contains 1 entry of size 0x38 (56 bytes). So it ends at offset 0x40 + 1*0x38 = 0x78 (this is the first byte after this table, and this is also where your "mysterious data" begins, so keep this in mind).
The Section Headers Table starts at offset 0x98 in the file and contains 3 entries of size 0x40 (64 bytes), that is, each entry in SHT takes 4 consecutive rows in a hex editor, and the entire table is 3*4 = 12 such rows, so the offset 0x158 is the first byte after this table. But this is just the end of the file, so there's nothing more after the SHT.
The SHT entry at index 2 (the third=last one) should be a string table that contains the names for the sections.
So let's look at those sections now, shall we?
Section #2
Let's start with section #2, since it is supposed to contain the string table with the names for all the sections, so it will be very useful in further analysis. Here's its header (the last one in the table):
Name index Type=SHT_STRTAB (bingo!)
Flags .----^----. .----^----.
00000118 .----------^----------. 01 00 00 00 03 00 00 00 |........|
00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000130 7e 00 00 00 00 00 00 00 11 00 00 00 00 00 00 00 |~...............|
'----------.----------' '----------.----------'
Starting offset Size
00000140 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000150 00 00 00 00 00 00 00 00 |........|
00000158
So this is indeed a string table (0x03 = SHT_STRTAB). It starts from offset 0x7E in the file and takes 0x11 (17) consecutive bytes. The first byte after the string table is therefore 0x8F. This byte is not a part of any section (garbage).
The string table
So let's see what's in the section containing the string table, so that we could name our sections:
0000007E 00 2e |..|
00000080 73 68 73 74 72 74 61 62 00 2e 74 65 78 74 00 |shstrtab..text.|
0000008F
Here's the string table, with addresses relative to its beginning:
+0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +A +B +C +D +E +F
00: 00 2E 73 68 73 74 72 74 61 62 00 2e 74 65 78 74
10: 00
or the same in ASCII, with the NULL characters marked as ∎:
+0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +A +B +C +D +E +F
00: ∎ . s h s t r t a b ∎ . t e x t
10: ∎
So we have just 3 full string in it, with the following relative offsets:
00: "" (Just the empty string)
01: ".shstrtab" (Name for this section)
0B: ".text" (Name for the section that contains the executable code)
(Keep in mind, though, that sections can also address substrings inside those strings, if they share the common ending.)
We can now verify that this section (#2) is indeed named .shstrtab: its name index was 0x01 after all, wasn't it? ;)
Section #1
Now let's take apart section #1's header:
Name index Type=SHT_PROGBITS
Flags .----^----. .----^----.
000000d8 .----------^----------. 0b 00 00 00 01 00 00 00 |........|
000000e0 06 00 00 00 00 00 00 00 78 00 40 00 00 00 00 00 |........x.#.....|
000000f0 78 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 |x...............|
'----------.----------' '----------.----------'
Starting offset Size
00000100 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000110 00 00 00 00 00 00 00 00 |........|
00000118
So this section is named .text (note the name index 0x0B) and it is of type SHT_PROGBITS, so it contains some program-defined data; the executable code in this case. It starts from the offset 0x78 in the file and takes the next 6 bytes, so the first byte after this section is at offset 0x7E (where the string table begins). Here's its contents:
00000070 31 c0 ff c0 cd 80 |1.....|
0000007E
But wait! Remember where your "mysterious data" starts? Yes! It's the 0x78 offset! :) So this "mysterious data" is actually your executable payload :) After decoding it as Intel x86-64 opcodes we get this tiny little program:
31 C0 xor %eax,%eax ; Clear the EAX register to 0 (the short way).
FF C0 inc %eax ; Increase the EAX, so now it contains 1.
CD 80 int $0x80 ; Interrupt 0x80 is the system call on Linux.
which is basically equivalent to calling exit(0) in C ;) because the syscall interrupt expects the operation number in EAX, which in this case is sys_exit (operation number 1).
So yeah, mystery solved :) But let's continue anyway, to learn something more, and this way we'll find out where this piece of code will be loaded in memory.
Section #0
And finally section #0. It has some part missing, but I assume it was all 0s, since the first section is always a NULL section after all. Here's its (butchered) header:
00000098 00 00 00 00 00 00 00 00 | ........|
*
000000d0 00 00 00 00 00 00 00 00
But it's of no use to us. Nothing interesting here.
Program Headers Table
The last thing what's left to decode is the Program Headers Table, which – according to the information from the ELF header – starts from the offset 0x40 and takes 56 bytes, the first byte after it being at offset 0x78. Here's the dump:
Type=PHT_EXEC Flags=RX Starting offset in file
.----^----. .----^----. .----------^----------.
00000040 01 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 40 00 00 00 00 00 00 00 40 00 00 00 00 00 |..#.......#.....|
'----------.----------' '----------.----------'
Virtual address Physical address
Size in file Size in memory
.----------^----------. .----------^----------.
00000060 7e 00 00 00 00 00 00 00 7e 00 00 00 00 00 00 00 |~.......~.......|
00000070 00 00 20 00 00 00 00 00
00000078 '----------.----------'
Alignment
So it says that we load the first 126 (0x7E) bytes of the file into a memory segment of the same size, and the memory segment is supposed to start from the virtual address 0x400000. Our code starts from the offset 0x78 in the file and the first byte after it has the offset 0x7E, so it basically loads the entire beginning of the file, with the ELF header and the program header table into memory, as well as our executable payload at the end of it, and stops loading afterwards, ignoring the rest of the file.
So if the beginning of the file is loaded at address 0x400000, and our program starts 120 (0x78) bytes from its beginning, it will be located at the address 0x400078 in memory :>
Now let's see what entry point is specified in the ELF header for our program:
Executable x86-64 Version=1 Program's entry point
.-^-. .-^-. .----^----. .----------^----------.
00000010 02 00 3e 00 01 00 00 00 78 00 40 00 00 00 00 00 |..>.....x.#.....|
Bingo! :> It's 0x400078, so it points at the start of our little piece of code in the memory image.
And that's all, folks! ;)
I've wiped all the CHS/LBA information and bootcode on a spare flash drive leaving only the bare sector count information:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80 ff |................|
000001c0 ff ff 0c ff ff ff 00 08 00 00 00 f8 d4 01 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
Leaving only this:
80 ff ff ff 0c ff ff ff 00 08 00 00 00 f8 d4 01
The first flag saying it is bootable, the 4th that it is FAT32, and the last 2, 4 byte values the sector start and count in Big-endian. The FF's are CHS/LBA info I set all to FF
And, the disk still works fine. Partition shows up, I can mount it, read from it, write etc.
I'm on Linux. How does Windows handle the LBA/CHS info? Is CHS/LBA irrelevant in this day and age?
The JPEG file signature states that the JPEG EOI (End of Image) marker is given by the final FF D9. However I very occasionally come across JPEGs that also have a small amount of data following this EOI (in this example 02 24 ...):
FF D8 FF E1
...
...
FF D9
02 24 02 24 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00 00 27 10 00 00 00 00 00 00 00 00 27 10 00 00 00 00 27 10 00 00 27 10 00 00 00 00 00 00 00 00 27 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 41 58 53 21 00 1E 5D 32 00 00 00 00
Does the JPEG standard allow for data following the EOI marker or is this additional binary data corruption?
If this is not corruption what information is encoded in these final bytes?
The JPEG standard only defines a JPEG data stream. That is what goes between the SOI and EOI markets. Anything outside that is outside JPEG.
JPEG does not define a file format so technically there is nothing wrong with having extra data at the begin or the end. However, most decoders will puke if you have extra data before the SOI marker.
First time I am trying any shell script, the task which I need to perform is below:
Out put of this command is below -> cat /sys/kernel/debug/spmi/spmi-0/data
00800 00 03 03 00 01 01 00 C0 10 00 00 00 00 20 00 00
00810 00 03 03 03 00 03 03 00 00 00 00 00 00 00 00 00
00820 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00830 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00840 0F 07 01 00 0F 07 04 00 0F 07 07 80 0F 07 04 00
00850 0F 07 04 00 0F 03 08 00 00 00 01 80 00 00 00 00
00860 00 00 00 80 00 00 04 80 00 00 04 00 00 00 00 00
00870 0F 00 00 00 02 04 00 00 00 00 00 00 00 00 00 00
00880 FE 00 40 00 00 00 00 00 05 00 20 00 01 00 00 00
I need to check the value of first row and 14th column value and extract it. That value either can be 00 or 20. On the basis of that value i have to change the dir name which i think i can take care.
can any body help me out in this as i have googled from https://unix.stackexchange.com/questions/37313/how-do-i-grep-for-multiple-patterns but could not make it.
It is very easy!
$ head /sys/kernel/debug/spmi/spmi-0/data -n 1| cut -d " " -f 14
Explanation:
head /sys/kernel/debug/spmi/spmi-0/data -n 1 it will output first line from given file.
cut -d " " -f 14 will select 14th field/column (where fields get delimited by " " -Space
Edit
Usage
value=`head /sys/kernel/debug/spmi/spmi-0/data -n 1| cut -d " " -f 14`
echo $value
awk - solution
Alternate solution provided by twalberg
awk 'NR==1{print $14}' /sys/kernel/debug/spmi/spmi-0/data
Create a pattern which you need to search like in your case it’s 00 or 20 as i understand.
pattern="00|20"
cat /sys/kernel/debug/spmi/spmi-0/data | grep -E “${pattern}”
Then you can use cut -d " " -f2 for getting specific colum
This will give the output based on your pattern, hope i understood your problem correct.
I'm looking for an easy way to convert a simple binary file into a text-representation of its binary, where encoding doesn't matter. I know that the programmatic solution is straightforward, but I feel that there must be some arcane string of unix commands to accomplish this.
Am I off base? Is there a simpler solution than the programmatic?
base64 -e filename>xxx
on the other side
base64 -d xxx>filename
Use od. For example:
$ od -t x1 -An /bin/ls | head
7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
02 00 3e 00 01 00 00 00 e0 26 40 00 00 00 00 00
40 00 00 00 00 00 00 00 30 b6 01 00 00 00 00 00
00 00 00 00 40 00 38 00 09 00 40 00 1d 00 1c 00
06 00 00 00 05 00 00 00 40 00 00 00 00 00 00 00
40 00 40 00 00 00 00 00 40 00 40 00 00 00 00 00
f8 01 00 00 00 00 00 00 f8 01 00 00 00 00 00 00
08 00 00 00 00 00 00 00 03 00 00 00 04 00 00 00
38 02 00 00 00 00 00 00 38 02 40 00 00 00 00 00
38 02 40 00 00 00 00 00 1c 00 00 00 00 00 00 00
for example, to display a binary file as a sequence of hex codes:
od -t x1 file|cut -c8-
uuencode and uudecode were made for transferring binary content as ASCII characters. See the wikipedia entry.
max#upsight:~$ openssl base64 < /dev/urandom | head -10
qnISxigXTjgON+tkSDtRJ6fRNczsejY2bEC5D1W8fscy+6mopiGfVLvZ/bu99SrT
qdTRaeRXO8fgEejXsbTy4XP9MmCbAsBCSEvDpq5bfR/Sd7EjJLUxcRwzEMlhIrYT
m6J+20aR9M4g7pbT+hjjBE/gsHKxFfZQFgxT/tm1pEg6zMvQywjsrc7d+PSJQOHw
vzYXfWkyLO1nJm9g+Pw3rBI/UuV0+lmrIflhlj5CDWuaxDJUXJiWdsD6cGKLclfz
Mlh17mHwteqMLLSrTZ0QA0ygxISqiCf2sDtPgUToM7ZT2EbaNck5auxbhU+7OcxI
vBZRKozRZtfsZA0IUzMlIQmFanBdjOeGepQjgCDruq5hqEbNc1A+HhXqTtAr8Aic
4iNf36xZifDvASYy27hTVrlI/5kTeRZURqquaxHqum15VD5IC3J/sH+AwPpN1/qi
0YM8xt+LliVje7Oo7QiTona+VMjA//a715/0J8yeryLxTLSnT8JsXUpR0CiOgAcH
tQk9nzHCfMmFzb02nrhFJ0MjLCFgNJOiI1vT0AhNnMh449dcIkDDwyMpkRV4KZ1l
CSL+K4vXhMz3LhPKSihKbYLY6aJSnlPe/GiIOfl1g1VlbtoxJ7ZclpcOp4KWSKHV
...and so on
If the reason you're doing it is to see strings inside the binary data then there's a command called "strings" that will print all the strings in a file for you.
you can also use hexdump. Look at the man page for more options
$ hexdump binaryfile
Yes, you are off-base, this is nontrivial in the general case. Some commercial solutions exist, one we use is Autonomy Keyview.
I am assuming you mean including (e.g.) MSOffice and PDFs.