grepping for a large binary value from an even larger binary file - linux

Question

As the title suggests I would like to grep a reasonably large (about 100MB) binary file, for a binary string - this binary string is just under 5K.
I've tried grep using the -P option, but this only seems to return matches when the pattern is only a few bytes - when I go up to about 100 bytes it no longer finds any matches.
I've also tried bgrep. This worked well originally, however, when I needed to extend the pattern to the length I have now I just get "invalid/empty search string" errors.
The irony is, in Windows I can use HxD to search the file and I finds it in a instance. What I really need though is a Linux command line tool.
Thanks for your help,
Simon

Answer 1

Say we have a couple of big binary data files. For a big one that shouldn't match, we create a 100MB file whose contents are all NUL bytes.
dd ibs=1 count=100M if=/dev/zero of=allzero.dat
For the one we want to match, create a hundred random megabytes.
#! /usr/bin/env perl
use warnings;
binmode STDOUT or die "$0: binmode: $!";
for (1 .. 100 * 1024 * 1024) {
print chr rand 256;
}
Execute it as ./mkrand >myfile.dat.
Finally, extract a known match into a file named pattern.
dd skip=42 count=10 if=myfile.dat of=pattern
I assume you want only the files that match (-l) and want your pattern to be treated literally (-F or --fixed-strings). I suspect you may have been running into a length limit with -P.
You may be tempted to use the --file=PATTERN-FILE option, but grep interprets the contents of PATTERN-FILE as newline-separated patterns, so in the likely case that your 5KB pattern contains newlines, you'll hit an encoding problem.
So hope your system's ARG_MAX is big enough and go for it. Be sure to quote the contents of pattern. For example:
$ grep -l --fixed-strings "$(cat pattern)" allzero.dat myfile.dat
myfile.dat

Answer 2

Try using grep -U which treats files as binary.
Also, how are you specifying the search pattern? It might just need escaping to survive shell parameter expansions

Answer 3

As the string you are searching is pretty long. You could benefit by an implementation of the Boyer-Moore search algorithm which is very efficient when search string is very long
http://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_search_algorithm
The wiki also has links to some sample code.

Answer 4

You might want to look at a simple Python script.
match= (b"..."
b"...."
b"..." ) # Some byte string literal of immense proportions
with open("some_big_file","rb") as source:
block= read(len(match))
while block != match:
byte= read(1)
if not byte: break
block= block[1:]+read(1)
This might work reliably under Linux as well as Windows.