Rsync's rolling checksum in haskell - haskell

Question

I'm starting learn both haskell and remote delta compression. My first step is to implement rsync's version of rolling checksum in haskell. Does chunk equal to X(i) in those formula? If so I'm getting confused.
Haskell can turn bytestring into byte array
How I do turn array of Word8s into that big chunk. Word32768? I mean what if X(i) is list of Word8s?
After that how to do arithmetical operation on those unsigned ints of 4KB size?
Also my current version of implementation just slides by 1B (Word8) each.

Answer 1

It's very easy to turn a ByteString into [Word8] using unpack, which should be sufficient for performing this algorithm (although not necessarily the most efficient).
Why do you need to turn Word8s into a Word32768? Why do you need a 2^15 bit number? That would be very difficult to represent, but you could use a list or array of Word8s, which is very easy to represent in memory and is equivalent.
For performing arithmetic, functions like map, zipWith, folds, and scans are very useful. For example, performing the first step of the algorithm:
import qualified Data.ByteString as BS
a :: Int -> Int -> ByteString -> Int
a k l x
= (`mod` m)
$ sum
$ map fromIntegral
$ take (l - k)
$ drop k
$ BS.unpack x
where m = 2 ^ 16
And implementing the function b is only marginally more difficult, you just have to figure out the sequence of l - i + 1 for i = k to l, then use zipWith (*) between the map fromIntegral and take (l - k). After that, it's quite trivial to implement s, although it certainly could be performed more efficiently if you factor out the common steps of take (l - k) $ drop k $ BS.unpack x.

Answer 2

Chunk does not equal to X(i) in that equation/formula in the link provided. It is mostly related to Data Deduplication. Also, rolling checksum can be used to create chunks, identify chunk boundaries etc.
Also my current implementation of rsync's rolling checksum is as follows. Next I'm gonna implement cyclic polynomial rolling checksum then read some books on Data Deduplication
import qualified Data.ByteString.Lazy as B
import qualified Data.ByteString.Lazy.Char8 as B8
import Data.Word
import Data.Bits
import Data.Int
type CheckSumPartial = Word16
type CheckSumA = CheckSumPartial
type CheckSumB = CheckSumPartial
type WindowSize = Int64
type CheckSum = Word32
type Byte = Word8
main:: IO ()
main = do
let str = B8.pack "abcdef"
let s1 = roll 3 str
let s2 = withoutRoll 3 str
print s1
print s2
return ()
roll :: WindowSize -> B.ByteString -> [CheckSum]
roll w str =
let
(a,b,s) = newABS w str
h = B.head str
t = B.tail str
in if fromIntegral (B.length t) < w
then [s]
else s : rollNext w t h a b
withoutRoll :: WindowSize -> B.ByteString -> [CheckSum]
withoutRoll w str =
let
(_,_,s) = newABS w str
t = B.tail str
in if fromIntegral (B.length t) < w
then [s]
else s : withoutRoll w t
newA :: WindowSize -> B.ByteString -> CheckSumA
newA w str =
let block = B.take w str
in B.foldr aSum (0::CheckSumA) block
where
aSum x acc = acc + (fromIntegral x :: CheckSumA)
newB :: WindowSize -> B.ByteString -> CheckSumB
newB w str =
let block = B.take w str
in fst $ B.foldr bSum (0::CheckSumB, w) block
where
bSum x (acc,l) = (acc + fromIntegral l * (fromIntegral x :: CheckSumB), l-1)
rollA :: CheckSumA -> Byte -> Byte -> CheckSumA
rollA prevA prevHead curLast = prevA - fromIntegral prevHead + fromIntegral curLast
rollB :: CheckSumA -> Byte -> WindowSize -> CheckSumB -> CheckSumB
rollB curA prevHead w prevB = prevB - fromIntegral w * fromIntegral prevHead + curA
calculateS :: CheckSumA -> CheckSumB -> CheckSum
calculateS a b = (fromIntegral a :: Word32) .|. shift (fromIntegral b :: Word32) 16
rollNext :: WindowSize ->B.ByteString -> Byte -> CheckSumA -> CheckSumB -> [CheckSum]
rollNext w str prevHead prevA prevB =
let
curBlock = B.take (fromIntegral w) str
curLast = B.last curBlock
h = B.head str
t = B.tail str
a = rollA prevA prevHead curLast
b = rollB a prevHead w prevB
s = calculateS a b
in if fromIntegral (B.length t) < w
then [s]
else s : rollNext w t h a b
newABS :: WindowSize -> B.ByteString -> (CheckSumA, CheckSumB, CheckSum)
newABS w str =
let a = newA w str
b = newB w str
s = calculateS a b
in (a,b,s)