Finding data in large binary file and output with context - linux

Question

Prologue / Context
Last week my root filesystem was remounted readonly serveral times and I took a complete snapshot via ddrescue. Sadly the filesystem was damaged already and some files are missing. At the moment I try to find my ejabberd user-database which should be somewhere within the image. Testdisk found the required file (marked as deleted) but could not restore it. Since the file is pretty small and I have a backup from some month ago I thought about doing a binary search over the whole image.
So now I have a 64GB file with a damaged filesystem and would like to extract some 4kb blocks which contain a certain pattern.
Question
How can I find the data within the 64GB large file and extract the result with some context (4kb)?
Since the filesystem image resides on my server I would prefer a linux cli tool.

Answer 1

The Tool
Since I couldn't find a tool which meet my requirements I wrote it myself in golang. I call it bima (for binary match). It isn't pretty but it did the job:
package main
import (
"bytes"
"encoding/hex"
"fmt"
"gopkg.in/alecthomas/kingpin.v1"
"io"
"log"
"math"
"os"
)
var (
debug = kingpin.Flag("debug", "Enable debug mode.").Short('d').Bool()
bsize = kingpin.Flag("blocksize", "Blocksize").Short('b').Default("126976").Int()
debugDetail = kingpin.Flag("debugdetail", "Debug Detail").Short('v').Default("10").Int()
matchCommand = kingpin.Command("match", "Match a value")
matchCommandValue = matchCommand.Arg("value", "The value (Hex Encoded e.g.: 616263 == abc)").Required().String()
matchCommandFile = matchCommand.Arg("file", "The file").Required().String()
)
func main() {
kingpin.Version("0.1")
mode := kingpin.Parse()
if *bsize <= 0 {
log.Fatal("The blocksize has to be larger than 0")
}
if *debugDetail <= 0 {
log.Fatal("The Debug Detail has to be larger than 0")
}
if mode == "match" {
searchBytes, err := hex.DecodeString(*matchCommandValue)
if err != nil {
log.Fatal(err)
}
scanFile(searchBytes, *matchCommandFile)
}
}
func scanFile(search []byte, path string) {
searchLength := len(search)
blocksize := *bsize
f, err := os.Open(path)
if err != nil {
log.Fatal(err)
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
log.Fatal(err)
}
filesize := fi.Size()
expectedRounds := int(math.Ceil(float64(filesize-int64(searchLength))/float64(blocksize)) + 1)
if expectedRounds <= 0 {
expectedRounds = 1
}
data := make([]byte, 0, blocksize+searchLength-1)
data2 := make([]byte, 0, blocksize+searchLength-1)
offset := make([]byte, searchLength-1)
//reading the len of the slice or less (but not the cap)
readCount, err := f.Read(offset)
if err == io.EOF {
fmt.Println("The files seems to be empty")
return
} else if err != nil {
log.Fatal(err)
}
data = append(data, offset...)
buffer := make([]byte, blocksize)
var blockpos int
var idx int
blockpos = 0
lastLevel := -1
roundLevel := 0
idxOffset := 0
for round := 0; ; round++ {
if *debug {
roundLevel = ((round * 100) / expectedRounds)
if (roundLevel%*debugDetail == 0) && (roundLevel > lastLevel) {
lastLevel = roundLevel
fmt.Fprintln(os.Stderr, "Starting round", round+1, "of", expectedRounds, "--", ((round * 100) / expectedRounds))
}
}
//At EOF, the count will be zero and err will be io.EOF
readCount, err = f.Read(buffer)
if err != nil {
if err == io.EOF {
if *debug {
fmt.Fprintln(os.Stderr, "Done - Found EOF")
}
break
}
fmt.Println(err)
return
}
data = append(data, buffer[:readCount]...)
data2 = data
idxOffset = 0
for {
idx = bytes.Index(data2, search)
if idx >= 0 {
fmt.Println(blockpos + idxOffset + idx)
if idx+searchLength < len(data2) {
data2 = data2[idx+searchLength:]
idxOffset += idx
} else {
break
}
} else {
break
}
}
data = data[readCount:]
blockpos += readCount
}
}
The Story
For completeness here comes what I did to solve my problem:
At first I used hexedit to find out, that all db files have the same header. Encoded in hex it looks like this: 0102030463584d0b0000004b62574c41
So I used my tool to find all occurrences within my sda.image file:
./bima match 0102030463584d0b0000004b62574c41 ./sda.image >DBfiles.txt
For the 64GB this took about 8 Minutes and I think the HDD was the limiting factor.
The result where about 1200 occurrences which I extracted from the image with dd. As I didn't know the exact size of the files I simply extracted chunks of 20.000 bytes:
for f in $(cat DBfiles.txt); do
dd if=sda.image of=$f.dunno bs=1 ibs=1 skip=$f count=20000
done
Now I had about 1200 files and had to find the right ones. In a first step I search for the passwd files (passwd.DCD and passwd.DCL). later I did the same for the roster files. As the header of the files contains the name, I simply greped for passwd:
for f in *.dunno; do
if [ "$(cat $f | head -c 200 | grep "passwd" | wc -l)" == "1" ]; then
echo "$f" | sed 's/\.$//g' >> passwd_files.list
fi
done
Because the chunks were larger than the files I had to find the end of each files manually. I did the corrections with Curses Hexedit.
During that process I could see that the head of each file contained either dcl_logk or dcd_logk. So I knew which of the files were DCL files and which were DCD files.
In the end I had each file up to ten times and had to decide which version I wanted to use. In general I took the largest file. After putting the files in the DB directory of the new ejabberd server and restarting it, all accounts are back again. :-)