Let's say that I have a string 5a.
This is the hex representation of the ASCII letter Z.
I need to find a Linux shell command which will take a hex string and output the ASCII characters that the hex string represents.
So if I do:
echo 5a | command_im_looking_for
I will see a solitary letter Z:
Z
I used to do this with xxd:
echo -n 5a | xxd -r -p
But then I realised that in Debian/Ubuntu, xxd is part of vim-common and hence might not be present in a minimal system. To also avoid Perl (IMHO also not part of a minimal system), I ended up using sed, xargs, and printf like this:
echo -n 5a | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf
Mostly, I only want to convert a few bytes and it's okay for such tasks. The advantage of this solution over the one of ghostdog74 is, that this can convert hex strings of arbitrary lengths automatically. xargs is used because printf doesnt read from standard input.
echo -n 5a | perl -pe 's/([0-9a-f]{2})/chr hex $1/gie'
Note that this won't skip non-hex characters. If you want just the hex (no whitespace from the original string etc):
echo 5a | perl -ne 's/([0-9a-f]{2})/print chr hex $1/gie'
Also, zsh and bash support this natively in echo:
echo -e '\x5a'
You can do this with echo only, without the other stuff. Don't forget to add "-n" or you will get a linebreak automatically:
echo -n -e "\x5a"
Bash one-liner
echo -n "5a" | while read -N2 code; do printf "\x$code"; done
Some Python 3 one-liners that work with any number of bytes.
Decoding hex
Using strip, so that it's ok to have a newline on stdin.
$ echo 666f6f0a | python3 -c "import sys, binascii; sys.stdout.buffer.write(binascii.unhexlify(input().strip()))"
foo
Encoding hex
$ echo foo | python3 -c "import sys, binascii; print(binascii.hexlify(sys.stdin.buffer.read()).decode())"
666f6f0a
Depending on where you got that "5a", you can just prepend "\x" to it and pass that to printf:
$ a=5a
$ a="\x${a}"
$ printf "$a"
Z
echo 5a | python -c "import sys; print chr(int(sys.stdin.read(),base=16))"
Here is a pure bash script (as printf is a bash builtin) :
#warning : spaces do matter
die(){ echo "$#" >&2;exit 1;}
p=48656c6c6f0a
test $((${#p} & 1)) == 0 || die "length is odd"
p2=''; for ((i=0; i<${#p}; i+=2));do p2=$p2\\x${p:$i:2};done
printf "$p2"
If bash is already running, this should be faster than any other solution which is launching a new process.
dc can convert between numeric bases:
$ echo 5a | (echo 16i; tr 'a-z' 'A-Z'; echo P) | dc
Z$
There is a simple shell command ascii.
If you use Ubuntu, install it with:
sudo apt install ascii
Then
ascii 0x5a
will output:
ASCII 5/10 is decimal 090, hex 5a, octal 132, bits 01011010: prints as `Z'
Official name: Majuscule Z
Other names: Capital Z, Uppercase Z
As per #Randal comment, you can use perl, e.g.
$ printf 5a5a5a5a | perl -lne 'print pack "H*", $_'
ZZZZ
and other way round:
$ printf ZZZZ | perl -lne 'print unpack "H*", $_'
5a5a5a5a
Another example with file:
$ printf 5a5a5a5a | perl -lne 'print pack "H*", $_' > file.bin
$ perl -lne 'print unpack "H*", $_' < file.bin
5a5a5a5a
You can use this command (python script) for larger inputs:
echo 58595a | python -c "import sys; import binascii; print(binascii.unhexlify(sys.stdin.read().strip()).decode())"
The result will be:
XYZ
And for more simplicity, define an alias:
alias hexdecoder='python -c "import sys; import binascii; print(binascii.unhexlify(sys.stdin.read().strip()).decode())"'
echo 58595a | hexdecoder
GNU awk 4.1
awk -niord '$0=chr("0x"RT)' RS=.. ORS=
Note that if you echo to this it will produce an extra null byte
$ echo 595a | awk -niord '$0=chr("0x"RT)' RS=.. ORS= | od -tx1c
0000000 59 5a 00
Y Z \0
Instead use printf
$ printf 595a | awk -niord '$0=chr("0x"RT)' RS=.. ORS= | od -tx1c
0000000 59 5a
Y Z
Also note that GNU awk produces UTF-8 by default
$ printf a1 | awk -niord '$0=chr("0x"RT)' RS=.. ORS= | od -tx1
0000000 c2 a1
If you are dealing with characters outside of ASCII, and you are going to be
Base64 encoding the resultant string, you can disable UTF-8 with -b
echo 5a | sha256sum | awk -bniord 'RT~/\w/,$0=chr("0x"RT)' RS=.. ORS=
Similar to my answer here: Linux shell scripting: hex number to binary string
You can do it with the same tool like this (using ascii printable character instead of 5a):
echo -n 616263 | cryptocli dd -decoders hex
Will produce the following result:
abcd
Related
The following is the content stored in my file
This is my Input
So, using wc -c command we can get the number of characters stored in the file.
My expected output for above file that edited by using Vim in Ubuntu is 16. But, wc -c command returns 17.
Why is the output like this? There isn't even a carriage return at end of line. So, what is the 17th character?
Of course you had enter. Maybe you can't see it. Consider these two examples:
echo -n "This is my Input" | wc -c
16
Because -n is for avoiding enter, but
echo "This is my Input" | wc -c
17
Look at this example too see the new line:
How to see newline?
echo "This is my Input" | od -c
od dumps files in octal and other formats. -c selects ASCII characters or backslash escapes.
And here is an example for file and usage of od:
In Linux, when Vim saves buffers, it will terminate every line by appending line terminator of new line.
You can open your file and input :!xxd to view hex-dump or directly use hexdump yourfile command.
0000000: 5468 6973 2069 7320 6d79 2049 6e70 7574 This is my Input
0000010: 0a .
~
~
~
In there you can see, the file have appended 0a in the end of file.
So when you use wc -c to get the number of this file, it will return 17 that includes the new line symbol.
The input string you are giving as input has no enter/new line, but echo is assigning enter/newline to it. And wc -c reads enter or newline from given by the echo command.
For example
echo k | wc -c
returns 2 because 1 for k and 1 for new line appended by echo.
While
echo -n k | wc -c
returns 1 because -n suppresses the newline.
But wc -c always reads newline.
You can try
printf k | wc -c
It returns 1.
See what it does in file:
bash-4.1$ echo 1234 > newfile
bash-4.1$ cat newfile
1234
bash-4.1$ cat -e newfile
1234$
bash-4.1$ printf 1234 > newfile
bash-4.1$ cat newfile
1234bash-4.1$ cat -e newfile
1234bash-4.1$
You have 17 because of the /0 chaeracter.
What's the way to swap first 4 chars with last 4 chars of string?
e.g. I have the string 20140613, I'd like to convert that to 06132014.
$ f=20140613
$ g=${f#????}${f%????}
$ echo $g
06132014
For dealing with longer strings something like the following is needed. (With inspiration from konsolebox's answer.)
echo ${f:(-4)}${f:4:${#f} - 8}${f:0:4}
Using pure BASH regex:
s='20140613'
[[ "$s" =~ ^(.*)([[:digit:]]{4})$ ]] && echo "${BASH_REMATCH[2]}${BASH_REMATCH[1]}"
06132014
Simply use substring expansion:
$ STRING=20140613
$ echo "${STRING:(-4)}${STRING:0:4}"
06132014
See Parameter Expansion.
Using date which is optimized for such kind of conversion:
$ str="20140613"
$ date +"%m%d%Y" -d "$str"
06132014
When you have to convert dates, no need to look so far ;)
Using sed:
STRING="20140613"
STRING=$(echo $STRING | sed 's/\(....\)\(.*\)/\2\1/')
Or using awk:
echo 20140613 | awk '{print substr($0,5,7) substr($0,1,4)}'
Test:
~$ echo 20140613 | awk '{print substr($0,5,7) substr($0,1,4)}'
>> 06132014
Through sed,
$ echo 20140613 | sed 's/^\(.\{4\}\)\(.\{4\}\)$/\2\1/g'
06132014
Through perl,
$ echo 20140613 | perl -pe 's/^(.{4})(.{4})$/\2\1/g'
06132014
With GNU Coreutils:
input=20140613
output=$(echo $input | fold -w4 | tac | tr -d \\n)
If you also need the last line feed, you can replace tr -d \\n with printf %s%s\\n or just append && echo to the command.
With perl
for str in 11112222 1111xxxx2222 111222
do
echo -n "$str -> "
echo "$str" | perl -ple 's/^(.{4})(.*)(.{4})$/\3\2\1/'
done
produces:
11112222 -> 22221111
1111xxxx2222 -> 2222xxxx1111
111222 -> 111222
I have binary and want to extract part of it, starting from know byte string (i.e. FF D8 FF D0) and ending with known byte string (AF FF D9)
In the past I've used dd to cut part of binary file from beginning/ending but this command doesn't seem to support what I ask.
What tool on terminal can do this?
Locate the start/end position, then extract the range.
$ xxd -g0 input.bin | grep -im1 FFD8FFD0 | awk -F: '{print $1}'
0000cb0
$ ^FFD8FFD0^AFFFD9^
0009590
$ dd ibs=1 count=$((0x9590-0xcb0+1)) skip=$((0xcb0)) if=input.bin of=output.bin
In a single pipe:
xxd -c1 -p file |
awk -v b="ffd8ffd0" -v e="aaffd9" '
found == 1 {
print $0
str = str $0
if (str == e) {found = 0; exit}
if (length(str) == length(e)) str = substr(str, 3)}
found == 0 {
str = str $0
if (str == b) {found = 1; print str; str = ""}
if (length(str) == length(b)) str = substr(str, 3)}
END{ exit found }' |
xxd -r -p > new_file
test ${PIPESTATUS[1]} -eq 0 || rm new_file
The idea is to use awk between two xxd to select the part of the file that is needed. Once the 1st pattern is found, awk prints the bytes until the 2nd pattern is found and exit.
The case where the 1st pattern is found but the 2nd is not must be taken into account. It is done in the END part of the awk script, which return a non-zero exit status. This is catch by bash's ${PIPESTATUS[1]} where I decided to delete the new file.
Note that en empty file also mean that nothing has been found.
This should work with standard tools (xxd, tr, grep, awk, dd). This correctly handles the "pattern split across line" issue, also look for the pattern only aligned at byte offset (not nibble).
file=<yourfile>
outfile=<youroutputfile>
startpattern="ff d8 ff d0"
endpattern="af ff d9"
xxd -g0 -c1 -ps ${file} | tr '\n' ' ' > ${file}.hex
start=$((($(grep -bo "${startpattern}" ${file}.hex\
| head -1 | awk -F: '{print $1}')-1)/3))
len=$((($(grep -bo "${endpattern}" ${file}.hex\
| head -1 | awk -F: '{print $1}')-1)/3-${start}))
dd ibs=1 count=${len} skip=${start} if=${file} of=${outfile}
Note: The script above use a temporary file to prevent having the binary>hex conversion twice. A space/time trade-off is to pipe the result of xxd directly into the two grep. A one-liner is also possible, at the expense of clarity.
One could also use tee and named pipe to prevent having to store a temporary file and converting output twice, but I'm not sure it would be faster (xxd is fast) and is certainly more complex to write.
See this link for a way to do binary grep. Once you have the start and end offset, you should be able with dd to get what you need.
A variation on the awk solution that assumes that your binary file, once converted in hex with spaces, fits in memory:
xxd -c1 -p file |
tr "\n" " " |
sed -n -e 's/.*\(ff d8 ff d0.*aa ff d9\).*/\1/p' |
xxd -r -p > new_file
Another solution in sed, but using less memory:
xxd -c1 -p file |
sed -n -e '1{N;N;N}' -e '/ff\nd8\nff\nd0/{:begin;p;s/.*//;n;bbegin}' -e 'N;D' |
sed -n -e '1{N;N}' -e '/aa\nff\nd9/{p;Q1}' -e 'P;N;D' |
xxd -r -p > new_file
test ${PIPESTATUS[2]} -eq 1 || rm new_file
The 1st sed prints from ff d8 ff d0 till the end of file. Note that you need as much N in -e '1{N;N;N}' as there is bytes in your 1st pattern less one.
The 2nd sed prints from the beginning of the file to aa ff d9. Note again that you need as much N in -e '1{N;N}' as there is bytes in your 2nd pattern less one.
Again, a test is needed to check if the 2nd pattern is found, and delete the file if it is not.
Note that the Q command is a GNU extension to sed. If you do not have it, you need to trash the rest of the file once the pattern is found (in a loop like the 1st sed, but not printing the file), and check after hex to binary conversion that the new_file end with the wright pattern.
For example given an input file like below:
sid|storeNo|latitude|longitude
2|1|-28.03õ720000
9|2
10
jgn
352|1|-28.03¿720000
9|2|fd¿kjhn422-405
000¥0543210|gf¿djk39
gfd|f¥d||fd
Output (the characters below can appear in any order):
¿õ¥
Does anyone have a function (awk, bash, perl.etc) that could scan each line and then output (in octal, hex or ascii - either is fine) a distinct list of the control characters (for simplicity, control characters being those above ascii char 126) found?
Using perl v5.8.8.
To print the bytes in octal:
perl -ne'printf "%03o\n", ord for /[^\x09\x0A\x20-\x7E]/g' file | sort -u
To print the bytes in hex:
perl -ne'printf "%02X\n", ord for /[^\x09\x0A\x20-\x7E]/g' file | sort -u
To print the original bytes:
perl -nE'say for /[^\x09\x0A\x20-\x7E]/g' file | sort -u
This should catch everything over ordinal value 126 without having to explicitly weed out outliers
#!/bin/bash
while IFS= read -n1 c; do
if (( $(printf "%d" "'$c") > 126)); then
echo "$c"
fi
done < ./infile | sort -u
Output
¥
¿
õ
To delete everything except the control characters:
tr -d '\0-\176' < input > output
To test:
printf 'foobar\n\377' | tr -d '\0-\176' | od -t c
See tr(1) man page for details.
sed -e 's/[A-Za-z0-9,|]//g' -e 's/-//g' -e 's/./&^M/g' | sort -u
Delete everything you don't want, put everything else on its own line, then sort -u the whole kit.
The "&^M" is "&" followed by Ctrl-V followed by Ctrl-M in Bash.
Unix wins.
I want to convert binary data to hexadecimal, just that, no fancy formatting and all. hexdump seems too clever, and it "overformats" for me. I want to take x bytes from the /dev/random and pass them on as hexadecimal.
Preferably I'd like to use only standard Linux tools, so that I don't need to install it on every machine (there are many).
Perhaps use xxd:
% xxd -l 16 -p /dev/random
193f6c54814f0576bc27d51ab39081dc
Watch out!
hexdump and xxd give the results in a different endianness!
$ echo -n $'\x12\x34' | xxd -p
1234
$ echo -n $'\x12\x34' | hexdump -e '"%x"'
3412
Simply explained. Big-endian vs. little-endian :D
With od (GNU systems):
$ echo abc | od -A n -v -t x1 | tr -d ' \n'
6162630a
With hexdump (BSD systems):
$ echo abc | hexdump -ve '/1 "%02x"'
6162630a
From Hex dump, od and hexdump:
"Depending on your system type, either or both of these two utilities will be available--BSD systems deprecate od for hexdump, GNU systems the reverse."
Perhaps you could write your own small tool in C, and compile it on-the-fly:
int main (void) {
unsigned char data[1024];
size_t numread, i;
while ((numread = read(0, data, 1024)) > 0) {
for (i = 0; i < numread; i++) {
printf("%02x ", data[i]);
}
}
return 0;
}
And then feed it from the standard input:
cat /bin/ls | ./a.out
You can even embed this small C program in a shell script using the heredoc syntax.
All the solutions seem to be hard to remember or too complex. I find using printf the shortest one:
$ printf '%x\n' 256
100
But as noted in comments, this is not what author wants, so to be fair, below is the full answer.
... to use above to output actual binary data stream:
printf '%x\n' $(cat /dev/urandom | head -c 5 | od -An -vtu1)
What it does:
printf '%x\n' .... - prints a sequence of integers , i.e. printf '%x,' 1 2 3, will print 1,2,3,
$(...) - this is a way to get output of some shell command and process it
cat /dev/urandom - it outputs random binary data
head -c 5 - limits binary data to 5 bytes
od -An -vtu1 - octal dump command, converts binary to decimal
As a testcase ('a' is 61 hex, 'p' is 70 hex, ...):
$ printf '%x\n' $(echo "apple" | head -c 5 | od -An -vtu1)
61
70
70
6c
65
Or to test individual binary bytes, on input let’s give 61 decimal ('=' char) to produce binary data ('\\x%x' format does it). The above command will correctly output 3d (decimal 61):
$printf '%x\n' $(echo -ne "$(printf '\\x%x' 61)" | head -c 5 | od -An -vtu1)
3d
If you need a large stream (no newlines) you can use tr and xxd (part of Vim) for byte-by-byte conversion.
head -c1024 /dev/urandom | xxd -p | tr -d $'\n'
Or you can use hexdump (POSIX) for word-by-word conversion.
head -c1024 /dev/urandom | hexdump '-e"%x"'
Note that the difference is endianness.
dd + hexdump will also work:
dd bs=1 count=1 if=/dev/urandom 2>/dev/null | hexdump -e '"%x"'
Sometimes perl5 works better for portability if you target more than one platform. It comes with every Linux distribution and Unix OS. You can often find it in container images where other tools like xxd or hexdump are not available. Here's how to do the same thing in Perl:
$ head -c8 /dev/urandom | perl -0777 -ne 'print unpack "H*"'
5c9ed169dabf33ab
$ echo -n $'\x01\x23\xff' | perl -0777 -ne 'print unpack "H*"'
0123ff
$ echo abc | perl -0777 -ne 'print unpack "H*"'
6162630a
Note that this uses slurp more, which causes Perl to read the entire input into memory, which may be suboptimal when the input is large.
These three commands will print the same (0102030405060708090a0b0c):
n=12
echo "$a" | xxd -l "$n" -p
echo "$a" | od -N "$n" -An -tx1 | tr -d " \n" ; echo
echo "$a" | hexdump -n "$n" -e '/1 "%02x"'; echo
Given that n=12 and $a is the byte values from 1 to 26:
a="$(printf '%b' "$(printf '\\0%o' {1..26})")"
That could be used to get $n random byte values in each program:
xxd -l "$n" -p /dev/urandom
od -vN "$n" -An -tx1 /dev/urandom | tr -d " \n" ; echo
hexdump -vn "$n" -e '/1 "%02x"' /dev/urandom ; echo