I have a kernel that the vendor hasn't provided the source for. It is the gziped kernel. Where does the data part of the sequence start? I tried to find the magic number (1f 8b) and copy that into a gzip file, but I can't decode it in 7zip.
You have the correct approach for a gzip-compressed binary. The decompression is different for burrows-wheeler (bzip2) or LZMA. If it doesn't decompress with 7zip, try using something like gzip/zcat.
An example of decompressing gzip-encoded kernels, based on Benjamin Coddington's post How to extract vmlinux from vmlinuz [archived from the original]:
$ mkdir -p /tmp/kernel-uncompressed/; cd /tmp/kernel-uncompressed/
$ f="vmlinuz-`uname -r`" # e.g. "vmlinuz-2.6.18-128.el5.uvm6PAE"
$ cp /boot/$f .
$ od -t x1 -A d $f | grep "1f 8b 08"
0008320 1b 00 1f 8b 08 00 d5 c2 9a 49 02 03 ec 3b 7d 7c
$ offset=8322 # Where the gzip marker starts, based on the above output.
$ dd bs=1 skip=$offset if=$f | zcat > vmlinux
Related
I am on Linux. I have received a mixed list of files, which I have forgotten to verify beforehand. My editor (emacs) has used LF (\n) for some files which originally had CR+LF (\r\n) (!!). I have realized about this way too late, and I think this is causing me trouble.
I would like to find all files in my cwd which have at least one CR+LF in them. I do not trust the file command, because I think it only checks the first lines, and not the whole file.
I would like to check whole files to look for CR + LF. Is there a tool for that, or do I need to roll my own?
You can use this grep command to list all the files in a directory with at least one CR-LF:
grep -l $'\r$' *
Pattern $'\r$' will file \r just before end of each line.
Or using hex value:
grep -l $'\x0D$' *
Where \x0D will find \r (ASCII: 13).
dos2unix can not only convert dos line ends (CR+LF) to unis (LF) but also display file information with a -i option. e.g.
sh-4.3$ (echo "1" ; echo "") > 123.txt
sh-4.3$ unix2dos 123.txt
unix2dos: converting file 123.txt to DOS format...
sh-4.3$ cat 123.txt ; hexdump -C 123.txt ; dos2unix --info='du' 123.txt
1
00000000 31 0d 0a 0d 0a |1....|
00000005
2 0 123.txt
sh-4.3$ dos2unix 123.txt
dos2unix: converting file 123.txt to Unix format...
sh-4.3$ cat 123.txt ; hexdump -C 123.txt ; dos2unix --info='du' 123.txt
1
00000000 31 0a 0a |1..|
00000003
0 2 123.txt
I´m trying to solve some security problem with file, whose MD5 hash is 76cdb2bad9582d23c1f6f4d868218d6c.
I don't have that file, but based on internet MD5 searches, I think, it is minimum size zip file.
I found one example and its content is exactly same like minimum size zip file.
Is it possible to create such minimum size zip file (22 B) with Linux (or Windows) command?
BTW: Recently, I solved similar task with gzip file: gzip -n EmptyFileName
Here you go:
50 4b 05 06 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00
That has the MD5 signature you provided.
If you have Info-ZIP's zip, you can create it thusly:
zip empty.zip anyfile
zip -d empty.zip anyfile
That adds "anyfile" to a new zip file, and then deletes it from the zip file, leaving it empty.
An easier-to-use version for copying-and-pasting into the shell:
echo UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA== | base64 -d > empty.zip
This just prints the base64'd version of the empty zip file (created by creating a zip file with a single file then deleting that single file from the zip file), and reverse the encoding with base64 -d and writes the output to empty.zip.
If the version of base64 that ships with your machine doesn't use the same syntax as above, here's a more-portable but less-terse alternative:
echo UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA== | openssl enc -d -base64 > empty.zip
With Info-Zip (standard zip/unzip installation on Ubuntu 20.04), you will have to use the -i "*" option. If it's not present, zip will complain and exit with error code 1 on creation. Counter-intuitively, its behaviour changes with -i "*".
mkdir /tmp/mytest
cd /tmp/mytest
zip -r mytest.zip . -i "*"
mytest.zip will be the empty zip file you desire.
with this .bat script you can create empty.zip file under windows:
#echo off
del /q /f empty.zip >nul 2>nul
certutil -decode "%~f0" empty.zip
-----BEGIN CERTIFICATE-----
UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA==
-----END CERTIFICATE-----
tldr;
use printf
$ printf '\x50\x4b\x05\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >empty.zip
$ md5sum empty.zip
76cdb2bad9582d23c1f6f4d868218d6c empty.zip
use base64
$ base64 -d >empty.zip <<'EOF'
UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA==
EOF
$ md5sum empty.zip
76cdb2bad9582d23c1f6f4d868218d6c empty.zip
Explanation
While the zip tool doesn't support the creation of an empty zip file, you can create a non-empty zip file and then remove its contents:
$ touch foo.txt
$ zip empty.zip foo.txt
adding: foo.txt (stored 0%)
$ zip -d empty.zip foo.txt
deleting: foo.txt
zip warning: zip file empty
Note how zipinfo returns an error when given the resulting empty.zip:
$ zipinfo empty.zip
Archive: empty.zip
Zip file size: 22 bytes, number of entries: 0
Empty zipfile.
$ echo $?
1
You can confirm that empty.zip has the expected bytes:
$ file empty.zip
empty.zip: Zip archive data (empty)
$ xxd empty.zip
00000000: 504b 0506 0000 0000 0000 0000 0000 0000 PK..............
00000010: 0000 0000 0000 ......
You can use printf to write those 22 bytes (in hexadecimal) directly into a file:
$ printf '\x50\x4b\x05\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >empty.zip
You can also turn those 22 bytes into 32 base64-encoded characters and use that:
$ base64 <empty.zip
UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA==
I'm doing a little script on bash, which shows the total size in mb, the number of files, the number of the folder and the name of folder.
I have almost everything except the size in mb
du -a -h | cut -d/ -f2 | sort | uniq -c
It shows something like this:
4 01 folder 01
6 02 folder 02
11 03 folder 03
13 04 folder 04
16 05 folder 05
.....
15 13 folder 13
1 5.7G .
as you see, the sort is: number of files, number of folder and name.
I want this:
300M 4 01 folder 01
435M 6 02 folder 02
690M 11 03 folder 03
780M 13 04 folder 04
1.6G 16 05 folder 05
.....
15 13 folder 13
1 5.7G .
thank you in advance.
PD there is some way to show the name over each column like this?
M F # name
300M 4 01 folder 01
435M 6 02 folder 02
690M 11 03 folder 03
780M 13 04 folder 04
1.6G 16 05 folder 05
.....
15 13 folder 13
1 5.7G .
How about this?
echo -e "Size\tFiles\tDirectory"; paste <(du -sh ./*/ | sort -k2 | cut -f1) <(find ./*/ | cut -d/ -f2 | uniq -c | sort -k2 | awk '{print ($1-1)"\t"$2}') | sort -nk2
Sample output:
Size Files Directory
172M 36 callrecords
17M 747 manual
83M 2251 input
7.5G 16867 output
Explanation:
Add the header:
echo -e "Size\tFiles\tDirectory";
<(COMMAND) is a structure which allows the output of a command to be used as if it were a file. Paste takes 2 files, and outputs them side by side. So we are pasting together the outputs of two commands. The first is this:
<(du -sh ./*/ | sort -k2 | cut -f1)
Which simply finds the size of subfolders of the current folder, summarising anything inside. This is then sorted according to the names of the files/folders, and then the first column is taken. This gives us a list of the sizes of subfolders of the current folder, sorted by their name.
The second command is this:
<(find ./*/ | cut -d/ -f2 | uniq -c | sort -k2 | awk '{print ($1-1)"\t"$2}')
This is similar to your original command - it finds folders below the current directory, truncates the names to include only the first sublevel, then counts them to give a list of sub-folders of the current folder, and the number of files within each. This is then sorted by the folder names, and the awk command formats the results and also subtracts 1 from the file count for each folder (as the folder itself is included). We can then paste the results together to get the (almost) final output.
Finally, we use sort -nk2 on the output of the paste command to sort by number on the 2nd field - ie the number of files.
I am trying to read from /dev/random and /dev/urandom and would like to know what is the best way to read from them and block/character special devices in general using bash shell scripting ?
Use dd to get blocks of data from the device. E.g. to get 8 bytes from /dev/urandom:
dd if=/dev/urandom count=1 bs=8 | ...
Then you can use od to convert the bytes to a human-readable form:
$ dd if=/dev/urandom count=1 bs=8 2>/dev/null | od -t x1 -A n
b4 bc 2f 59 dd 55 1b 4a
By the way, if you only need random numbers in bash, $RANDOM is probably more useful:
$ echo $RANDOM $RANDOM $RANDOM $RANDOM
3466 6521 4426 9349
My hint:
dd if=/dev/urandom count=4 | ...
or e.g. The tail is heavily dependent on what you want to do with that data
To format as a long integer number:
dd if=/dev/urandom bs=1 count=4|od -l
echo -n abcd > my_test_file
hexdump my_test_file
# 0000000 6261 6463
# 0000004
hexdump -C my_test_file
# 00000000 61 62 63 64 |abcd|
# 00000004
Is the file stored on harddisk like the first output and the "hexdump -C"-output is reordered for readability?
The second hexdump represents the storage on disk. The first variant exists more for historic reasons (16 bit big-endian architecture on PDP-11).