I have binary and want to extract part of it, starting from know byte string (i.e. FF D8 FF D0) and ending with known byte string (AF FF D9)
In the past I've used dd to cut part of binary file from beginning/ending but this command doesn't seem to support what I ask.
What tool on terminal can do this?
Locate the start/end position, then extract the range.
$ xxd -g0 input.bin | grep -im1 FFD8FFD0 | awk -F: '{print $1}'
0000cb0
$ ^FFD8FFD0^AFFFD9^
0009590
$ dd ibs=1 count=$((0x9590-0xcb0+1)) skip=$((0xcb0)) if=input.bin of=output.bin
In a single pipe:
xxd -c1 -p file |
awk -v b="ffd8ffd0" -v e="aaffd9" '
found == 1 {
print $0
str = str $0
if (str == e) {found = 0; exit}
if (length(str) == length(e)) str = substr(str, 3)}
found == 0 {
str = str $0
if (str == b) {found = 1; print str; str = ""}
if (length(str) == length(b)) str = substr(str, 3)}
END{ exit found }' |
xxd -r -p > new_file
test ${PIPESTATUS[1]} -eq 0 || rm new_file
The idea is to use awk between two xxd to select the part of the file that is needed. Once the 1st pattern is found, awk prints the bytes until the 2nd pattern is found and exit.
The case where the 1st pattern is found but the 2nd is not must be taken into account. It is done in the END part of the awk script, which return a non-zero exit status. This is catch by bash's ${PIPESTATUS[1]} where I decided to delete the new file.
Note that en empty file also mean that nothing has been found.
This should work with standard tools (xxd, tr, grep, awk, dd). This correctly handles the "pattern split across line" issue, also look for the pattern only aligned at byte offset (not nibble).
file=<yourfile>
outfile=<youroutputfile>
startpattern="ff d8 ff d0"
endpattern="af ff d9"
xxd -g0 -c1 -ps ${file} | tr '\n' ' ' > ${file}.hex
start=$((($(grep -bo "${startpattern}" ${file}.hex\
| head -1 | awk -F: '{print $1}')-1)/3))
len=$((($(grep -bo "${endpattern}" ${file}.hex\
| head -1 | awk -F: '{print $1}')-1)/3-${start}))
dd ibs=1 count=${len} skip=${start} if=${file} of=${outfile}
Note: The script above use a temporary file to prevent having the binary>hex conversion twice. A space/time trade-off is to pipe the result of xxd directly into the two grep. A one-liner is also possible, at the expense of clarity.
One could also use tee and named pipe to prevent having to store a temporary file and converting output twice, but I'm not sure it would be faster (xxd is fast) and is certainly more complex to write.
See this link for a way to do binary grep. Once you have the start and end offset, you should be able with dd to get what you need.
A variation on the awk solution that assumes that your binary file, once converted in hex with spaces, fits in memory:
xxd -c1 -p file |
tr "\n" " " |
sed -n -e 's/.*\(ff d8 ff d0.*aa ff d9\).*/\1/p' |
xxd -r -p > new_file
Another solution in sed, but using less memory:
xxd -c1 -p file |
sed -n -e '1{N;N;N}' -e '/ff\nd8\nff\nd0/{:begin;p;s/.*//;n;bbegin}' -e 'N;D' |
sed -n -e '1{N;N}' -e '/aa\nff\nd9/{p;Q1}' -e 'P;N;D' |
xxd -r -p > new_file
test ${PIPESTATUS[2]} -eq 1 || rm new_file
The 1st sed prints from ff d8 ff d0 till the end of file. Note that you need as much N in -e '1{N;N;N}' as there is bytes in your 1st pattern less one.
The 2nd sed prints from the beginning of the file to aa ff d9. Note again that you need as much N in -e '1{N;N}' as there is bytes in your 2nd pattern less one.
Again, a test is needed to check if the 2nd pattern is found, and delete the file if it is not.
Note that the Q command is a GNU extension to sed. If you do not have it, you need to trash the rest of the file once the pattern is found (in a loop like the 1st sed, but not printing the file), and check after hex to binary conversion that the new_file end with the wright pattern.
Related
My command:
awk 'NR==FNR{a[$0]=1;next;} substr($0,50,6) in a' file1 file2
The problem is that file 2 contains \000 characters and awk consider it as binary file.
Replacing \000 with space character:
tr '\000' ' ' < file2 > file2_not_binary
solves binary file problem.
However my file2 is a 20GB file. And I don't want to do tr separately and save result as another file. I want to pass the result of tr to awk.
I have tried:
awk 'NR==FNR{a[$0]=1;next;} substr($0,50,6) in a' file1 < (tr '\000' ' ' < file2)
But the result is:
The system cannot find the file specified.
Another question is: can my memory or awk handle such a big file at once? I'm working on 12GB RAM PC.
EDIT
One of the answer works as I expected (credits to Ed Morton)
tr '\000' ' ' < file2 | awk 'NR==FNR{a[$0];next} substr($0,50,6) in a' file1 -
However it is like 2 time slower then doing the same in 2 steps - first removing \000 and save it and then using awk to search. How I can speed it up?
EDIT2
My bad. Ed Morton solution is actually a little bit faster then doing the same in two separately commands.
Two commands separately: 08:37:053
Two commands piped: 08:07:204
Since awk isn't storing your 2nd file in memory the size of that file is irrelevant except for speed of execution. Try this:
tr '\000' ' ' < file2 | awk 'NR==FNR{a[$0];next} substr($0,50,6) in a' file1 -
It should be:
awk ... <(tr -d '\0' < file2)
# -------^ no space!
Check the manual about Process Substitution.
You could replace it in awk using gsub(/\000/," "). Testing, let's make a test file:
$ awk 'BEGIN{print "a b\000c d"}' > foo
$ hexdump -C foo
00000000 61 20 62 00 63 20 64 0a |a b.c d.|
00000008
And then:
$ awk '{print; gsub(/\000/," "); print}' foo
a bc d
a b c d
I figured out how to get the line number of the last matching word in the file :
cat -n textfile.txt | grep " b " | tail -1 | cut -f 1
It gave me the value of 1787. So, I passed it manually to the sed command to search for the lines that contains the sentence "blades are down" after that line number and it returned all the lines successfully
sed -n '1787,$s/blades are down/&/p' myfile.txt
Is there a way that I can pass the line number from the first command to the second one through a variable or a file so I can but them in the script to be executed automatically ?
Thank you.
You can do this by just connecting your two commands with xargs. 'xargs -I %' allows you to take the stdin from a previous command and place it whenever you want in the next command. The '%' is where your '1787' will be written:
cat -n textfile.txt | grep " b " | tail -1 | cut -f 1 | xargs -I % sed -n %',$s/blades are down/&/p' myfile.txt
You can use:
command substitution to capture the result of the first command in a variable.
simple string concatenation to use the variable in your sed comand
startLine=$(grep -n ' b ' textfile.txt | tail -1 | cut -d ':' -f1)
sed -n ${startLine}',$s/blades are down/&/p' myfile.txt
You don't strictly need the intermediate variable - you could simply use:
sed $(grep -n ' b ' textfile.txt | tail -1 | cut -d ':' -f1)',$s/blades are down/&/p' myfile.txt`
but it may make sense to do error checking on the result of the command substitution first.
Note that I've streamlined the first command by using grep's -n option, which puts the line number separated with : before each match.
First we can get "half" of the file after the last match of string2, then you can use grep to match all the string1
tac your_file | awk '{ if (match($0, "string2")) { exit ; } else {print;} }' | \
grep "string1"
but the order is reversed if you don't care about the order. But if you do care, just add another tac at the end with a pipe |.
This might work for you (GNU sed):
sed -n '/\n/ba;/ b /h;//!H;$!d;x;//!d;s/$/\n/;:a;/\`.*blades are down.*$/MP;D' file
This reads through the file storing all lines following the last match of the first string (" b ") in the hold space.
At the end of file, it swaps to the hold space, checks that it does indeed have at least one match, then prints out those lines that match the second string ("blades are down").
N.B. it makes the end case (/\n/) possible by adding a new line to the end of the hold space, which will eventually be thrown away. This also caters for the last line edge condition.
For example given an input file like below:
sid|storeNo|latitude|longitude
2|1|-28.03õ720000
9|2
10
jgn
352|1|-28.03¿720000
9|2|fd¿kjhn422-405
000¥0543210|gf¿djk39
gfd|f¥d||fd
Output (the characters below can appear in any order):
¿õ¥
Does anyone have a function (awk, bash, perl.etc) that could scan each line and then output (in octal, hex or ascii - either is fine) a distinct list of the control characters (for simplicity, control characters being those above ascii char 126) found?
Using perl v5.8.8.
To print the bytes in octal:
perl -ne'printf "%03o\n", ord for /[^\x09\x0A\x20-\x7E]/g' file | sort -u
To print the bytes in hex:
perl -ne'printf "%02X\n", ord for /[^\x09\x0A\x20-\x7E]/g' file | sort -u
To print the original bytes:
perl -nE'say for /[^\x09\x0A\x20-\x7E]/g' file | sort -u
This should catch everything over ordinal value 126 without having to explicitly weed out outliers
#!/bin/bash
while IFS= read -n1 c; do
if (( $(printf "%d" "'$c") > 126)); then
echo "$c"
fi
done < ./infile | sort -u
Output
¥
¿
õ
To delete everything except the control characters:
tr -d '\0-\176' < input > output
To test:
printf 'foobar\n\377' | tr -d '\0-\176' | od -t c
See tr(1) man page for details.
sed -e 's/[A-Za-z0-9,|]//g' -e 's/-//g' -e 's/./&^M/g' | sort -u
Delete everything you don't want, put everything else on its own line, then sort -u the whole kit.
The "&^M" is "&" followed by Ctrl-V followed by Ctrl-M in Bash.
Unix wins.
I want to convert binary data to hexadecimal, just that, no fancy formatting and all. hexdump seems too clever, and it "overformats" for me. I want to take x bytes from the /dev/random and pass them on as hexadecimal.
Preferably I'd like to use only standard Linux tools, so that I don't need to install it on every machine (there are many).
Perhaps use xxd:
% xxd -l 16 -p /dev/random
193f6c54814f0576bc27d51ab39081dc
Watch out!
hexdump and xxd give the results in a different endianness!
$ echo -n $'\x12\x34' | xxd -p
1234
$ echo -n $'\x12\x34' | hexdump -e '"%x"'
3412
Simply explained. Big-endian vs. little-endian :D
With od (GNU systems):
$ echo abc | od -A n -v -t x1 | tr -d ' \n'
6162630a
With hexdump (BSD systems):
$ echo abc | hexdump -ve '/1 "%02x"'
6162630a
From Hex dump, od and hexdump:
"Depending on your system type, either or both of these two utilities will be available--BSD systems deprecate od for hexdump, GNU systems the reverse."
Perhaps you could write your own small tool in C, and compile it on-the-fly:
int main (void) {
unsigned char data[1024];
size_t numread, i;
while ((numread = read(0, data, 1024)) > 0) {
for (i = 0; i < numread; i++) {
printf("%02x ", data[i]);
}
}
return 0;
}
And then feed it from the standard input:
cat /bin/ls | ./a.out
You can even embed this small C program in a shell script using the heredoc syntax.
All the solutions seem to be hard to remember or too complex. I find using printf the shortest one:
$ printf '%x\n' 256
100
But as noted in comments, this is not what author wants, so to be fair, below is the full answer.
... to use above to output actual binary data stream:
printf '%x\n' $(cat /dev/urandom | head -c 5 | od -An -vtu1)
What it does:
printf '%x\n' .... - prints a sequence of integers , i.e. printf '%x,' 1 2 3, will print 1,2,3,
$(...) - this is a way to get output of some shell command and process it
cat /dev/urandom - it outputs random binary data
head -c 5 - limits binary data to 5 bytes
od -An -vtu1 - octal dump command, converts binary to decimal
As a testcase ('a' is 61 hex, 'p' is 70 hex, ...):
$ printf '%x\n' $(echo "apple" | head -c 5 | od -An -vtu1)
61
70
70
6c
65
Or to test individual binary bytes, on input let’s give 61 decimal ('=' char) to produce binary data ('\\x%x' format does it). The above command will correctly output 3d (decimal 61):
$printf '%x\n' $(echo -ne "$(printf '\\x%x' 61)" | head -c 5 | od -An -vtu1)
3d
If you need a large stream (no newlines) you can use tr and xxd (part of Vim) for byte-by-byte conversion.
head -c1024 /dev/urandom | xxd -p | tr -d $'\n'
Or you can use hexdump (POSIX) for word-by-word conversion.
head -c1024 /dev/urandom | hexdump '-e"%x"'
Note that the difference is endianness.
dd + hexdump will also work:
dd bs=1 count=1 if=/dev/urandom 2>/dev/null | hexdump -e '"%x"'
Sometimes perl5 works better for portability if you target more than one platform. It comes with every Linux distribution and Unix OS. You can often find it in container images where other tools like xxd or hexdump are not available. Here's how to do the same thing in Perl:
$ head -c8 /dev/urandom | perl -0777 -ne 'print unpack "H*"'
5c9ed169dabf33ab
$ echo -n $'\x01\x23\xff' | perl -0777 -ne 'print unpack "H*"'
0123ff
$ echo abc | perl -0777 -ne 'print unpack "H*"'
6162630a
Note that this uses slurp more, which causes Perl to read the entire input into memory, which may be suboptimal when the input is large.
These three commands will print the same (0102030405060708090a0b0c):
n=12
echo "$a" | xxd -l "$n" -p
echo "$a" | od -N "$n" -An -tx1 | tr -d " \n" ; echo
echo "$a" | hexdump -n "$n" -e '/1 "%02x"'; echo
Given that n=12 and $a is the byte values from 1 to 26:
a="$(printf '%b' "$(printf '\\0%o' {1..26})")"
That could be used to get $n random byte values in each program:
xxd -l "$n" -p /dev/urandom
od -vN "$n" -An -tx1 /dev/urandom | tr -d " \n" ; echo
hexdump -vn "$n" -e '/1 "%02x"' /dev/urandom ; echo
Let's say that I have a string 5a.
This is the hex representation of the ASCII letter Z.
I need to find a Linux shell command which will take a hex string and output the ASCII characters that the hex string represents.
So if I do:
echo 5a | command_im_looking_for
I will see a solitary letter Z:
Z
I used to do this with xxd:
echo -n 5a | xxd -r -p
But then I realised that in Debian/Ubuntu, xxd is part of vim-common and hence might not be present in a minimal system. To also avoid Perl (IMHO also not part of a minimal system), I ended up using sed, xargs, and printf like this:
echo -n 5a | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf
Mostly, I only want to convert a few bytes and it's okay for such tasks. The advantage of this solution over the one of ghostdog74 is, that this can convert hex strings of arbitrary lengths automatically. xargs is used because printf doesnt read from standard input.
echo -n 5a | perl -pe 's/([0-9a-f]{2})/chr hex $1/gie'
Note that this won't skip non-hex characters. If you want just the hex (no whitespace from the original string etc):
echo 5a | perl -ne 's/([0-9a-f]{2})/print chr hex $1/gie'
Also, zsh and bash support this natively in echo:
echo -e '\x5a'
You can do this with echo only, without the other stuff. Don't forget to add "-n" or you will get a linebreak automatically:
echo -n -e "\x5a"
Bash one-liner
echo -n "5a" | while read -N2 code; do printf "\x$code"; done
Some Python 3 one-liners that work with any number of bytes.
Decoding hex
Using strip, so that it's ok to have a newline on stdin.
$ echo 666f6f0a | python3 -c "import sys, binascii; sys.stdout.buffer.write(binascii.unhexlify(input().strip()))"
foo
Encoding hex
$ echo foo | python3 -c "import sys, binascii; print(binascii.hexlify(sys.stdin.buffer.read()).decode())"
666f6f0a
Depending on where you got that "5a", you can just prepend "\x" to it and pass that to printf:
$ a=5a
$ a="\x${a}"
$ printf "$a"
Z
echo 5a | python -c "import sys; print chr(int(sys.stdin.read(),base=16))"
Here is a pure bash script (as printf is a bash builtin) :
#warning : spaces do matter
die(){ echo "$#" >&2;exit 1;}
p=48656c6c6f0a
test $((${#p} & 1)) == 0 || die "length is odd"
p2=''; for ((i=0; i<${#p}; i+=2));do p2=$p2\\x${p:$i:2};done
printf "$p2"
If bash is already running, this should be faster than any other solution which is launching a new process.
dc can convert between numeric bases:
$ echo 5a | (echo 16i; tr 'a-z' 'A-Z'; echo P) | dc
Z$
There is a simple shell command ascii.
If you use Ubuntu, install it with:
sudo apt install ascii
Then
ascii 0x5a
will output:
ASCII 5/10 is decimal 090, hex 5a, octal 132, bits 01011010: prints as `Z'
Official name: Majuscule Z
Other names: Capital Z, Uppercase Z
As per #Randal comment, you can use perl, e.g.
$ printf 5a5a5a5a | perl -lne 'print pack "H*", $_'
ZZZZ
and other way round:
$ printf ZZZZ | perl -lne 'print unpack "H*", $_'
5a5a5a5a
Another example with file:
$ printf 5a5a5a5a | perl -lne 'print pack "H*", $_' > file.bin
$ perl -lne 'print unpack "H*", $_' < file.bin
5a5a5a5a
You can use this command (python script) for larger inputs:
echo 58595a | python -c "import sys; import binascii; print(binascii.unhexlify(sys.stdin.read().strip()).decode())"
The result will be:
XYZ
And for more simplicity, define an alias:
alias hexdecoder='python -c "import sys; import binascii; print(binascii.unhexlify(sys.stdin.read().strip()).decode())"'
echo 58595a | hexdecoder
GNU awk 4.1
awk -niord '$0=chr("0x"RT)' RS=.. ORS=
Note that if you echo to this it will produce an extra null byte
$ echo 595a | awk -niord '$0=chr("0x"RT)' RS=.. ORS= | od -tx1c
0000000 59 5a 00
Y Z \0
Instead use printf
$ printf 595a | awk -niord '$0=chr("0x"RT)' RS=.. ORS= | od -tx1c
0000000 59 5a
Y Z
Also note that GNU awk produces UTF-8 by default
$ printf a1 | awk -niord '$0=chr("0x"RT)' RS=.. ORS= | od -tx1
0000000 c2 a1
If you are dealing with characters outside of ASCII, and you are going to be
Base64 encoding the resultant string, you can disable UTF-8 with -b
echo 5a | sha256sum | awk -bniord 'RT~/\w/,$0=chr("0x"RT)' RS=.. ORS=
Similar to my answer here: Linux shell scripting: hex number to binary string
You can do it with the same tool like this (using ascii printable character instead of 5a):
echo -n 616263 | cryptocli dd -decoders hex
Will produce the following result:
abcd