Say I have 8b1f 0008 0231 49f6 0300 f1f3 75f4 0c72 f775 0850 7676 720c 560d 75f0 02e5 ce00 0861 1302 0000 0000, how can I easily get a binary file from that without copying+pasting into a hex editor?
Use:
% xxd -r -p in.txt out.bin
See xxd.
This version will work with binary format too:
cat /bin/sh \
| od -A n -v -t x1 \
| tr -d '\r' \
| xxd -r -g 1 -p1 \
| md5sum && md5sum /bin/sh
The extra '\r' is just if you're dealing with DOS text files...
And process byte by byte to prevent endianness difference if running parts of a pipe on different systems.
All the present answers refer to the convenient xxd -r approach, but for situations where xxd is not available or convenient here is a more portable (and more flexible, but more verbose and less efficient) solution, using only POSIX shell syntax (it also compensates for odd-number of digits in input):
un_od() {
printf -- "$(
tr -d '\t\r\n ' | sed -e 's/^(.(.{2})*)$/0\1/' -e 's/\(.\{2\}\)/\\x\1/g'
)"
}
By the way: you don't specify whether your input is big-endian or little-endian, or whether you want big/little-endian output. Usually input such as in your question would be big-endian/network-order (e.g., as created by od -t x1 -An -v), and would be expected to transform to big-endian output. I presume xxd just assumes that default if not told otherwise, and this solution does that too. If byte-swapping is needed, how you do the byte-swapping also depends on the word-size of the system (e.g., 32 bit, 64 bit) and very rarely the byte-size (you can almost always assume 8-bit bytes - octets - though).
The below functions use a more complex version of the binary -> od -> binary trick to portably byteswap binary data, conditional on system endianness, and accounting for system word-size. The algorithm works for anything up to 72-bit word size (because seq -s '' 10 -> 12345678910 doesn't work):
if { sed --version 2>/dev/null || :; } | head -n 1 | grep -q 'GNU sed'; then
_sed() { sed -r "${#}"; }
else
_sed() { sed -E "${#}"; }
fi
sys_bigendian() {
return $(
printf 'I' | od -t o2 | head -n 1 | \
_sed -e 's/^[^ \t]+[ \t]+([^ \t]+)[ \t]*$/\1/' | cut -c 6
)
}
sys_word_size() { expr $(getconf LONG_BIT) / 8; }
byte_swap() {
_wordsize=$1
od -An -v -t o1 | _sed -e 's/^[ \t]+//' | tr -s ' ' '\n' | \
paste -d '\\' $(for _cnt in $(seq $_wordsize); do printf -- '- '; done) | \
_sed -e 's/^/\\/' -e '$ s/\\+$//' | \
while read -r _word; do
_thissize=$(expr $(printf '%s' "$_word" | wc -c) / 4)
printf '%s' "$(seq -s '' $_thissize)" | tr -d '\n' | \
tr "$(seq -s '' $_thissize -1 1)" "$_word"
done
unset _wordsize _prefix _word _thissize
}
You can use the above to output file contents in big-endian format regardless of system endianness:
if sys_bigendian; then
cat /bin/sh
else
cat /bin/sh | byte_swap $(sys_word_size)
fi
Here is the way to reverse "od" output:
echo "test" | od -A x -t x1 | sed -e 's|^[0-f]* ?||g' | xxd -r
test
Related
I have a list of base64 strings in a file (file.txt) that I need to convert into hex. E.g.,
6IwwfX8Cctn85LW+vItMhw==
wIsNfYESR9Nfueo7mg3f7Q==
A+MxnRyu6kotbKPZglQ0Fg==
Jt5jNIphpmfGoFgtgM7/Sg==
sN+Q0Xcu6JHlkqdhJlM/tw==
Command:
echo -n 6IwwfX8Cctn85LW+vItMhw== | base64 -d | od -t x1 -An
This command works individually (albeit the spaces in between), but I need to convert through each string in the file, which has more than 500 lines.
Basically, I want the above base64 string format to be decoded to the below example hex string format:
30aa268d130fb78a4f8cb6f300e4c760
Is there a way that I can call each line in the file (like a for each command) and pipe with the base64 command to convert? Any help is appreciated.
Try:
for b64 in $(cat e.txt); do echo "$b64" | base64 -d | od -t x1 -An | tr -d ' '; done
The tr -d ' ' at the end deletes all spaces.
cat file.txt | while read input ; do echo -n "$input" | base64 -d | od -t x1 -An ; done
cat file.txt | while read b64
do echo -n "$b64" | base64 -d | od -t x1 -An | sed 's/[\t ]*//g'
done
You can use Awk for this and run a command on the contents of the file using the getline() syntax
awk '{ cmd = "printf '%s' "$1 "| base64 -d | od -t x1 -An" }
{ while ( ( cmd | getline result ) > 0 ) { gsub(/[[:space:]]+/,"",result); $0 = result }; close(cmd) }1' file.txt
Use a temporary file for overwriting the file contents and move back the original file
tmpfile=$(mktemp)
awk '{ cmd = "printf '%s' "$1 "| base64 -d | od -t x1 -An" }
{ while ( ( cmd | getline result ) > 0 ) { gsub(/[[:space:]]+/,"",result); $0 = result }; close(cmd) }1' file.txt > "$tmpfile" && mv "$tmpfile" file.txt
This is a code that shows my all user names.
-q user | grep -A 0 -B 2 -e uid:\ 5'[0-9][0-9]' | grep ^name | cut -d " " -f2-
For example, the output is like...
usernameone
hello
whoami
Then, I hope that I want to check a length of all user names.
Like this output...
11 //usernameone
5 //hello
6 //whoami
How can I get a length of pipeline code?
Given some command cmd that produces the list of users, you can do this pretty easily with xargs:
$ cat x
usernameone
hello
whoami
$ cat x | xargs -L 1 sh -c 'printf "%s //%s\n" "$(echo -n "$1" | wc -c)" "$1"' '{}'
11 //usernameone
5 //hello
6 //whoami
To get a piped command might not be possible, so here's a one liner that uses a split and a while loop to accomplish this:
-q user | grep -A 0 -B 2 -e uid:\ 5'[0-9][0-9]' | grep ^name | cut -d " " -f2-|tr " " "\n"|while read user; do echo $(echo $user|wc -c) '//'$user;done|tr "\n" " ";echo
This should give you an output in the desired format. I used user as a file hence the cat
i=0;for token in $(cat user); do echo -n "${#token} //$token";echo;i=$((i+1));done;echo;
When I am trying to run the below Script it says invalid option 3 for cat..Whats the problem?
I am tried to use index file which specifies which file is ham and which is spam...to read the files and train spamfilter
#!bin/bash
DirBogoDict=$1
BogoFilter=/home/gunna/Downloads/bogofilter-1.2.4/src/bogofilter
x=0
for i in 'cat index | fgrep spam | head -300 | awk -F "/" '{print$2"/"$3}''
do
x=$((x+1)) ; echo $x
cat /home/gunna/Downloads/db-6.1.19.NC/build_unix/ceas08-1/$i| $BogoFilter -d $DirBogoDict -M -k 1024 -s
done
for i in 'cat index | fgrep ham | head -300 | awk -F "/" '{print$2"/"$3}''
do
x=$((x+1)) ; echo $x
cat /home/gunna/Downloads/db-6.1.19.NC/build_unix/ceas08-1/$i | $BogoFilter -d $DirBogoDict -M -k 1024 -n
done
This part
'cat index | fgrep spam | head -300 | awk -F "/" '{print$2"/"$3}''
needs to be in back-ticks, not single quotes
`cat index | fgrep spam | head -300 | awk -F "/" '{print$2"/"$3}'`
And you could probably simplify it a little with
for i in `fgrep spam index | head -300 | awk "/" '{print$2"/"$3}'`
Kdopen has explained the error you got , here is the improved code for similar for-loop function.
DirBogoDict=$1
BogoFilter=/home/gunna/Downloads/bogofilter-1.2.4/src/bogofilter
awk '/spam/&&++myctr<=300{print $2 FS $3}' FS="/" index |while read i
do
cat /home/gunna/Downloads/db-6.1.19.NC/build_unix/ceas08-1/"$i"| $BogoFilter -d ${DirBogoDict} -M -k 1024 -s
done
awk '/ham/&&++myctr<=300{print $2 FS $3}' FS="/" index |while read i
do
cat /home/gunna/Downloads/db-6.1.19.NC/build_unix/ceas08-1/"$i"| $BogoFilter -d ${DirBogoDict} -M -k 1024 -s
done
Also look at your file names , since cat is giving an error and an option is invalid. To demonstrate this, Let say you have a file a name -3error
executing the following command
cat -3error
will gave
cat: invalid option -- '3'
cat therefore is thinking the "-" is followed by one of its command line arguments. As a result you probably get an invalid option error.
I am trying to get the max version number from a directory where i have several versions of one program
for example if output of ls is
something01_1.sh
something02_0.1.2.sh
something02_0.1.sh
something02_1.1.sh
something02_1.2.sh
something02_2.0.sh
something02_2.1.sh
something02_2.3.sh
something02_3.1.2.sh
something.sh
I am getting the max version number with the following -
ls somedir | grep some_prefix | cut -d '_' -f2 | sort -t '.' -k1 -r | head -n 1
Now if at the same time i want to check it with the version number which i already have in the system, whats the best way to do it...
in bash i got this working (if 2.5 is the current version)
(ls somedir | grep some_prefix | cut -d '_' -f2; echo 2.5) | sort -t '.' -k1 -r | head -n 1
is there any other correct way to do it?
EDIT: In the above example some_prefix is something02.
EDIT: Actual Problem here is
(ls smthing; echo more) | sort
is it the best way to merge output of two commands/program for piping into third.
I have found the solution. The best way it seems is using process substitution.
cat <(ls smthing) <(echo more) | sort
for my version example
cat <(ls somedir | grep some_prefix | cut -d '_' -f2) <(echo 2.5) | sort -t '.' -k1 -r | head -n 1
for the benefit of future readers, I recommend - please drop the lure of one-liner and use glob as chepner suggested.
Almost similar question is asked on superuser.
more info about process substitution.
Is the following code more suitable to what you're looking for:
#/bin/bash
highest_version=$(ls something* | sort -V | tail -1 | sed "s/something02_\|\.sh//g")
current_version=$(echo $0 | sed "s/something02_\|\.sh//g")
if [ $current_version > $highest_version ]; then
echo "Uh oh! Looks like we need to update!";
fi
You can try something like this :
#! /bin/bash
lastversion() { # prefix
local prefix="$1" a=0 b=0 c=0 r f vmax=0
for f in "$prefix"* ; do
test -f "$f" || continue
read a b c r <<< $(echo "${f#$prefix} 0 0 0" | tr -C '[0-9]' ' ')
v=$(((a*100+b)*100+c))
if ((v>vmax)); then vmax=$v; fi
done
echo $vmax
}
lastversion "something02"
It will print: 30102
I want to convert binary data to hexadecimal, just that, no fancy formatting and all. hexdump seems too clever, and it "overformats" for me. I want to take x bytes from the /dev/random and pass them on as hexadecimal.
Preferably I'd like to use only standard Linux tools, so that I don't need to install it on every machine (there are many).
Perhaps use xxd:
% xxd -l 16 -p /dev/random
193f6c54814f0576bc27d51ab39081dc
Watch out!
hexdump and xxd give the results in a different endianness!
$ echo -n $'\x12\x34' | xxd -p
1234
$ echo -n $'\x12\x34' | hexdump -e '"%x"'
3412
Simply explained. Big-endian vs. little-endian :D
With od (GNU systems):
$ echo abc | od -A n -v -t x1 | tr -d ' \n'
6162630a
With hexdump (BSD systems):
$ echo abc | hexdump -ve '/1 "%02x"'
6162630a
From Hex dump, od and hexdump:
"Depending on your system type, either or both of these two utilities will be available--BSD systems deprecate od for hexdump, GNU systems the reverse."
Perhaps you could write your own small tool in C, and compile it on-the-fly:
int main (void) {
unsigned char data[1024];
size_t numread, i;
while ((numread = read(0, data, 1024)) > 0) {
for (i = 0; i < numread; i++) {
printf("%02x ", data[i]);
}
}
return 0;
}
And then feed it from the standard input:
cat /bin/ls | ./a.out
You can even embed this small C program in a shell script using the heredoc syntax.
All the solutions seem to be hard to remember or too complex. I find using printf the shortest one:
$ printf '%x\n' 256
100
But as noted in comments, this is not what author wants, so to be fair, below is the full answer.
... to use above to output actual binary data stream:
printf '%x\n' $(cat /dev/urandom | head -c 5 | od -An -vtu1)
What it does:
printf '%x\n' .... - prints a sequence of integers , i.e. printf '%x,' 1 2 3, will print 1,2,3,
$(...) - this is a way to get output of some shell command and process it
cat /dev/urandom - it outputs random binary data
head -c 5 - limits binary data to 5 bytes
od -An -vtu1 - octal dump command, converts binary to decimal
As a testcase ('a' is 61 hex, 'p' is 70 hex, ...):
$ printf '%x\n' $(echo "apple" | head -c 5 | od -An -vtu1)
61
70
70
6c
65
Or to test individual binary bytes, on input let’s give 61 decimal ('=' char) to produce binary data ('\\x%x' format does it). The above command will correctly output 3d (decimal 61):
$printf '%x\n' $(echo -ne "$(printf '\\x%x' 61)" | head -c 5 | od -An -vtu1)
3d
If you need a large stream (no newlines) you can use tr and xxd (part of Vim) for byte-by-byte conversion.
head -c1024 /dev/urandom | xxd -p | tr -d $'\n'
Or you can use hexdump (POSIX) for word-by-word conversion.
head -c1024 /dev/urandom | hexdump '-e"%x"'
Note that the difference is endianness.
dd + hexdump will also work:
dd bs=1 count=1 if=/dev/urandom 2>/dev/null | hexdump -e '"%x"'
Sometimes perl5 works better for portability if you target more than one platform. It comes with every Linux distribution and Unix OS. You can often find it in container images where other tools like xxd or hexdump are not available. Here's how to do the same thing in Perl:
$ head -c8 /dev/urandom | perl -0777 -ne 'print unpack "H*"'
5c9ed169dabf33ab
$ echo -n $'\x01\x23\xff' | perl -0777 -ne 'print unpack "H*"'
0123ff
$ echo abc | perl -0777 -ne 'print unpack "H*"'
6162630a
Note that this uses slurp more, which causes Perl to read the entire input into memory, which may be suboptimal when the input is large.
These three commands will print the same (0102030405060708090a0b0c):
n=12
echo "$a" | xxd -l "$n" -p
echo "$a" | od -N "$n" -An -tx1 | tr -d " \n" ; echo
echo "$a" | hexdump -n "$n" -e '/1 "%02x"'; echo
Given that n=12 and $a is the byte values from 1 to 26:
a="$(printf '%b' "$(printf '\\0%o' {1..26})")"
That could be used to get $n random byte values in each program:
xxd -l "$n" -p /dev/urandom
od -vN "$n" -An -tx1 /dev/urandom | tr -d " \n" ; echo
hexdump -vn "$n" -e '/1 "%02x"' /dev/urandom ; echo