echo -n abcd > my_test_file
hexdump my_test_file
# 0000000 6261 6463
# 0000004
hexdump -C my_test_file
# 00000000 61 62 63 64 |abcd|
# 00000004
Is the file stored on harddisk like the first output and the "hexdump -C"-output is reordered for readability?
The second hexdump represents the storage on disk. The first variant exists more for historic reasons (16 bit big-endian architecture on PDP-11).
Related
I am on Linux. I have received a mixed list of files, which I have forgotten to verify beforehand. My editor (emacs) has used LF (\n) for some files which originally had CR+LF (\r\n) (!!). I have realized about this way too late, and I think this is causing me trouble.
I would like to find all files in my cwd which have at least one CR+LF in them. I do not trust the file command, because I think it only checks the first lines, and not the whole file.
I would like to check whole files to look for CR + LF. Is there a tool for that, or do I need to roll my own?
You can use this grep command to list all the files in a directory with at least one CR-LF:
grep -l $'\r$' *
Pattern $'\r$' will file \r just before end of each line.
Or using hex value:
grep -l $'\x0D$' *
Where \x0D will find \r (ASCII: 13).
dos2unix can not only convert dos line ends (CR+LF) to unis (LF) but also display file information with a -i option. e.g.
sh-4.3$ (echo "1" ; echo "") > 123.txt
sh-4.3$ unix2dos 123.txt
unix2dos: converting file 123.txt to DOS format...
sh-4.3$ cat 123.txt ; hexdump -C 123.txt ; dos2unix --info='du' 123.txt
1
00000000 31 0d 0a 0d 0a |1....|
00000005
2 0 123.txt
sh-4.3$ dos2unix 123.txt
dos2unix: converting file 123.txt to Unix format...
sh-4.3$ cat 123.txt ; hexdump -C 123.txt ; dos2unix --info='du' 123.txt
1
00000000 31 0a 0a |1..|
00000003
0 2 123.txt
I´m trying to solve some security problem with file, whose MD5 hash is 76cdb2bad9582d23c1f6f4d868218d6c.
I don't have that file, but based on internet MD5 searches, I think, it is minimum size zip file.
I found one example and its content is exactly same like minimum size zip file.
Is it possible to create such minimum size zip file (22 B) with Linux (or Windows) command?
BTW: Recently, I solved similar task with gzip file: gzip -n EmptyFileName
Here you go:
50 4b 05 06 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00
That has the MD5 signature you provided.
If you have Info-ZIP's zip, you can create it thusly:
zip empty.zip anyfile
zip -d empty.zip anyfile
That adds "anyfile" to a new zip file, and then deletes it from the zip file, leaving it empty.
An easier-to-use version for copying-and-pasting into the shell:
echo UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA== | base64 -d > empty.zip
This just prints the base64'd version of the empty zip file (created by creating a zip file with a single file then deleting that single file from the zip file), and reverse the encoding with base64 -d and writes the output to empty.zip.
If the version of base64 that ships with your machine doesn't use the same syntax as above, here's a more-portable but less-terse alternative:
echo UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA== | openssl enc -d -base64 > empty.zip
With Info-Zip (standard zip/unzip installation on Ubuntu 20.04), you will have to use the -i "*" option. If it's not present, zip will complain and exit with error code 1 on creation. Counter-intuitively, its behaviour changes with -i "*".
mkdir /tmp/mytest
cd /tmp/mytest
zip -r mytest.zip . -i "*"
mytest.zip will be the empty zip file you desire.
with this .bat script you can create empty.zip file under windows:
#echo off
del /q /f empty.zip >nul 2>nul
certutil -decode "%~f0" empty.zip
-----BEGIN CERTIFICATE-----
UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA==
-----END CERTIFICATE-----
tldr;
use printf
$ printf '\x50\x4b\x05\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >empty.zip
$ md5sum empty.zip
76cdb2bad9582d23c1f6f4d868218d6c empty.zip
use base64
$ base64 -d >empty.zip <<'EOF'
UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA==
EOF
$ md5sum empty.zip
76cdb2bad9582d23c1f6f4d868218d6c empty.zip
Explanation
While the zip tool doesn't support the creation of an empty zip file, you can create a non-empty zip file and then remove its contents:
$ touch foo.txt
$ zip empty.zip foo.txt
adding: foo.txt (stored 0%)
$ zip -d empty.zip foo.txt
deleting: foo.txt
zip warning: zip file empty
Note how zipinfo returns an error when given the resulting empty.zip:
$ zipinfo empty.zip
Archive: empty.zip
Zip file size: 22 bytes, number of entries: 0
Empty zipfile.
$ echo $?
1
You can confirm that empty.zip has the expected bytes:
$ file empty.zip
empty.zip: Zip archive data (empty)
$ xxd empty.zip
00000000: 504b 0506 0000 0000 0000 0000 0000 0000 PK..............
00000010: 0000 0000 0000 ......
You can use printf to write those 22 bytes (in hexadecimal) directly into a file:
$ printf '\x50\x4b\x05\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >empty.zip
You can also turn those 22 bytes into 32 base64-encoded characters and use that:
$ base64 <empty.zip
UEsFBgAAAAAAAAAAAAAAAAAAAAAAAA==
I have two files and I want to see if the first 40 bytes are similar. How can I do this using hex dump?
If you are using the BSD hexdump utility (which will also be installed as hd, with a different default output format) then you can supply the -n40 command line parameter to limit the dump to the first 40 bytes:
hexdump -n40 filename
If you are using the Posix standard od, you need a capital N. You might find the following invocation useful:
od -N40 -w40 -tx1 -Ax filename
(You can do that with hexdump, too, but the format string is more work to figure out :) ).
Try this:
head -c 40 myfile | hexdump
Not sure why you need hexdump here,
diff <(dd bs=1 count=40 if=file1) <(dd bs=1 count=40 if=file2)
with hexdump:
diff <(dd bs=1 count=40 if=file1|hexdump) <(dd bs=1 count=40 if=file2|hexdump)
I have a kernel that the vendor hasn't provided the source for. It is the gziped kernel. Where does the data part of the sequence start? I tried to find the magic number (1f 8b) and copy that into a gzip file, but I can't decode it in 7zip.
You have the correct approach for a gzip-compressed binary. The decompression is different for burrows-wheeler (bzip2) or LZMA. If it doesn't decompress with 7zip, try using something like gzip/zcat.
An example of decompressing gzip-encoded kernels, based on Benjamin Coddington's post How to extract vmlinux from vmlinuz [archived from the original]:
$ mkdir -p /tmp/kernel-uncompressed/; cd /tmp/kernel-uncompressed/
$ f="vmlinuz-`uname -r`" # e.g. "vmlinuz-2.6.18-128.el5.uvm6PAE"
$ cp /boot/$f .
$ od -t x1 -A d $f | grep "1f 8b 08"
0008320 1b 00 1f 8b 08 00 d5 c2 9a 49 02 03 ec 3b 7d 7c
$ offset=8322 # Where the gzip marker starts, based on the above output.
$ dd bs=1 skip=$offset if=$f | zcat > vmlinux
I am trying to read from /dev/random and /dev/urandom and would like to know what is the best way to read from them and block/character special devices in general using bash shell scripting ?
Use dd to get blocks of data from the device. E.g. to get 8 bytes from /dev/urandom:
dd if=/dev/urandom count=1 bs=8 | ...
Then you can use od to convert the bytes to a human-readable form:
$ dd if=/dev/urandom count=1 bs=8 2>/dev/null | od -t x1 -A n
b4 bc 2f 59 dd 55 1b 4a
By the way, if you only need random numbers in bash, $RANDOM is probably more useful:
$ echo $RANDOM $RANDOM $RANDOM $RANDOM
3466 6521 4426 9349
My hint:
dd if=/dev/urandom count=4 | ...
or e.g. The tail is heavily dependent on what you want to do with that data
To format as a long integer number:
dd if=/dev/urandom bs=1 count=4|od -l