How can I get nth byte of ByteString in Haskell?
I tried to find function like !! for ByteStrings, but found nothing.
ByteString.index is the function you're looking for.
Most of the "containerish" types emulate the extended list interface; you also want to be careful because that index function will crash the program if you feed it a string that's too short (as will !! on ordinary lists). A better implementation might be
import Data.ByteString as B
nthByte :: Int -> B.ByteString -> Maybe Word8
nthByte n bs = fst <$> B.uncons (B.drop n bs)
which, reading inside out, drops the first n bytes (maybe producing an empty byte string), then attempts to split the first character from the remainder, and if successful, ignores the rest of the string.
Related
Looking at the put instances of the various ByteString types we find that the length of the bytestring is always prefixed in the binary file before writing it. For example here - https://hackage.haskell.org/package/binary-0.8.8.0/docs/src/Data.Binary.Class.html#put
Taking an example
instance Binary B.ByteString where
put bs = put (B.length bs) -- Why this??
<> putByteString bs
get = get >>= getByteString
Is there any particular reason for doing this? And is the only way to write Bytestring without prefixing the length - creating our own newtype wrapper and having an instance for Binary?
Is there any particular reason for doing this?
The idea of get and put is that you can combine several objects. For example you can write:
write_func :: ByteString -> Char -> Put
write_func some_bytestring some_char = do
put some_bytestring
put some_char
then you want to define a function that can read the data back, and evidently you want the two functions to act together as an identity function: that if the writer writes a certain ByteString and a certain Char, then you want the read function to read the same ByteString and character.
The reader function should look similar to:
read_fun :: Get (ByteString, Char)
read_fun = do
bs <- get
c <- get
return (bs, c)
but the problem is, when does a ByteString ends? The 'A' character could also be part of the ByteString. You thus need to somehow indicate where the ByteString ends. This can be done by saving the length, or some marker at the end. In case of a marker, you will need to "escape" the bytestring, such that it can not contain the marker itself.
But you thus need some mechanism to specify that when the ByteString ends.
And is the only way to write Bytestring without prefixing the length - creating our own newtype wrapper and having an instance for Binary?
No, in fact it is already in the instance definition. If you want to write a ByteString without length, then you can use putByteString :: ByteString -> Put:
write_func :: ByteString -> Char -> Put
write_func some_bytestring some_char = do
putByteString some_bytestring
put some_char
but when reading the ByteString back, you will need to figure out how many bytes you have to read.
I have a ByteString that is containing the representation of Floats. Each Float is represented by 3 bytes in the ByteString.
I need to do some processing on the Float values, so I would like to perform that processing on an Vector of Float values. What would be the best way to do this?
I have a function toFloat :: [Word8] -> Float that converts 3 bytes of the ByteString to a Float. So I was thinking of iterating over the ByteString in steps of 3 Bytes and converting every step to a Float for a vector.
I've looked at the library functions for Vector but I can't find anything that suits this purpose. Data.Vector.Storable.ByteString.byteStringToVector looked promising but it converts every byte (instead of every 3 bytes) and doesn't give me any control over how the conversion of ByteString to Float should happen.
Just use Data.Vector.generate:
V.generate (BS.length bs `div` 3) $ \i ->
myToFloat (bs BS.! 3*i) (bs BS.! 3*i+1) (bs BS.! 3*i+2)
It'll allocate the vector all at once, and populate it. Data.ByteString.! is O(1), so this is quite efficient.
Try using
splitAt :: Int -> ByteString -> (ByteString, ByteString)
to split the ByteString into two: one of exactly 3 characters, and another containing the rest of the input. You can use this to implement a recursive function that will give you all the groups of length 3 (similar to Data.List.Split.chunksOf), and then you can use unpack on each to get the [Word8] you need. Pass that through your toFloat function, and convert to a vector with Vector.fromList.
There are a number of steps there that seem like perhaps they could be expensive, but I think probably the compiler is smart enough to fuse some of them, like the unpack/fromList pair. And splitting a ByteString is O(1), so that part's not as expensive as it looks either. Seems like this ought to be as suitable an approach as any.
{-# LANGUAGE OverloadedStrings #-}
import Data.Attoparsec.Text
import Control.Applicative(many)
import Data.Word
parseManyNumbers :: Parser [Int] -- I'd like many to return a Vector instead
parseManyNumbers = many (decimal <* skipSpace)
main :: IO ()
main = print $ parseOnly parseManyNumbers "131 45 68 214"
The above is just an example, but I need to parse a large amount of primitive values in Haskell and need to use arrays instead of lists. This is something that possible in the F#'s Fparsec, so I've went as far as looking at Attoparsec's source, but I can't figure out a way to do it. In fact, I can't figure out where many from Control.Applicative is defined in the base Haskell library. I thought it would be there as that is where documentation on Hackage points to, but no such luck.
Also, I am having trouble deciding what data structure to use here as I can't find something as convenient as a resizable array in Haskell, but I would rather not use inefficient tree based structures.
An option to me would be to skip Attoparsec and implement an entire parser inside the ST monad, but I would rather avoid it except as a very last resort.
There is a growable vector implementation in Haskell, which is based on the great AMT algorithm: "persistent-vector". Unfortunately, the library isn't that much known in the community so far. However to give you a clue about the performance of the algorithm, I'll say that it is the algorithm that drives the standard vector implementations in Scala and Clojure.
I suggest you implement your parser around that data-structure under the influence of the list-specialized implementations. Here the functions are, btw:
-- | One or more.
some :: f a -> f [a]
some v = some_v
where
many_v = some_v <|> pure []
some_v = (fmap (:) v) <*> many_v
-- | Zero or more.
many :: f a -> f [a]
many v = many_v
where
many_v = some_v <|> pure []
some_v = (fmap (:) v) <*> many_v
Some ideas:
Data Structures
I think the most practical data structure to use for the list of Ints is something like [Vector Int]. If each component Vector is sufficiently long (i.e. has length 1k) you'll get good space economy. You'll have
to write your own "list operations" to traverse it, but you'll avoid re-copying data that you would have to perform to return the data in a single Vector Int.
Also consider using a Dequeue instead of a list.
Stateful Parsing
Unlike Parsec, Attoparsec does not provide for user state. However, you
might be able to make use of the runScanner function (link):
runScanner :: s -> (s -> Word8 -> Maybe s) -> Parser (ByteString, s)
(It also returns the parsed ByteString which in your case may be problematic since it will be very large. Perhaps you can write an alternate version which doesn't do this.)
Using unsafeFreeze and unsafeThaw you can incrementally fill in a Vector. Your s data structure might look
something like:
data MyState = MyState
{ inNumber :: Bool -- True if seen a digit
, val :: Int -- value of int being parsed
, vecs :: [ Vector Int ] -- past parsed vectors
, v :: Vector Int -- current vector we are filling
, vsize :: Int -- number of items filled in current vector
}
Maybe instead of a [Vector Int] you use a Dequeue (Vector Int).
I imagine, however, that this approach will be slow since your parsing function will get called for every single character.
Represent the list as a single token
Parsec can be used to parse a stream of tokens, so how about writing
your own tokenizer and letting Parsec create the AST.
The key idea is to represent these large sequences of Ints as a single token. This gives you a lot more latitude in how you parse them.
Defer Conversion
Instead of converting the numbers to Ints at parse time, just have parseManyNumbers return a ByteString and defer the conversion until
you actually need the values. This much enable you to avoid reifying
the values as an actual list.
Vectors are arrays, under the hood. The tricky thing about arrays is that they are fixed-length. You pre-allocate an array of a certain length, and the only way of extending it is to copy the elements into a larger array.
This makes linked lists simply better at representing variable-length sequences. (It's also why list implementations in imperative languages amortise the cost of copying by allocating arrays with extra space and copying only when the space runs out.) If you don't know in advance how many elements there are going to be, your best bet is to use a list (and perhaps copy the list into a Vector afterwards using fromList, if you need to). That's why many returns a list: it runs the parser as many times as it can with no prior knowledge of how many that'll be.
On the other hand, if you happen to know how many numbers you're parsing, then a Vector could be more efficient. Perhaps you know a priori that there are always n numbers, or perhaps the protocol specifies before the start of the sequence how many numbers there'll be. Then you can use replicateM to allocate and populate the vector efficiently.
I have a list of strings:
[" ix = index"," ctr = counter"," tbl = table"]
and I want to create a tuple from it like:
[("ix","index"),("ctr","counter"),("tbl","table")]
I even tried:
genTuple [] = []
genTuples (a:as)= do
i<-splitOn '=' a
genTuples as
return i
Any help would be appriciated
Thank you.
Haskell's type system is really expressive, so I suggest to think about the problem in terms of types. The advantage of this is that you can solve the problem 'top-down' and the whole program can be typechecked as you go, so you can catch all kinds of errors early on. The general approach is to incrementally divide the problem into smaller functions, each of which remaining undefined initially but with some plausible type.
What you want is a function (let's call it convert) which take a list of strings and generates a list of tuples, i.e.
convert :: [String] -> [(String, String)]
convert = undefined
It's clear that each string in the input list will need to be parsed into a 2-tuple of strings. However, it's possible that the parsing can fail - the sheer type String makes no guarantees that your input string is well formed. So your parse function maybe returns a tuple. We get:
parse :: String -> Maybe (String, String)
parse = undefined
We can immediately plug this into our convert function using mapMaybe:
convert :: [String] -> [(String, String)]
convert list = mapMaybe parse list
So far, so good - but parse is literally still undefined. Let's say that it should first verify that the input string is 'valid', and if it is - it splits it. So we'll need
valid :: String -> Bool
valid = undefined
split :: String -> (String, String)
split = undefined
Now we can define parse:
parse :: String -> Maybe (String, String)
parse s | valid s = Just (split s)
| otherwise = Nothing
What makes a string valid? Let's say it has to contain a = sign:
valid :: String -> Bool
valid s = '=' `elem` s
For splitting, we'll take all the characters up to the first = for the first tuple element, and the rest for the second. However, you probably want to trim leading/trailing whitespace as well, so we'll need another function. For now, let's make it a no-op
trim :: String -> String
trim = id
Using this, we can finally define
split :: String -> (String, String)
split s = (trim a, trim (tail b))
where
(a, b) = span (/= '=') s
Note that we can safely call tail here because we know that b is never empty because there's always a separator (that's what valid verified). Type-wise, it would've been nice to express this guarantee using a "non-empty string" but that may be a bit overengineered. :-)
Now, there are a lot of solutions to the problem, this is just one example (and there are ways to shorten the code using eta reduction or existing libraries). The main point I'm trying to get across is that Haskell's type system allows you to approach the problem in a way which is directed by types, which means the compiler helps you fleshing out a solution from the very beginning.
You can do it like this:
import Control.Monda
import Data.List
import Data.List.Split
map ((\[a,b] -> (a,b)) . splitOn "=" . filter (/=' ')) [" ix = index"," ctr = counter"," tbl = table"]
I know I already have the Haskell Data.ByteString.Lazy function to split a CSV on a single character, such as:
split :: Word8 -> ByteString -> [ByteString]
But I want to split on a multi-character ByteString (like splitting on a String instead of a Char):
split :: ByteString -> ByteString -> [ByteString]
I have multi-character separators in a csv-like text file that I need to parse, and the individual characters themselves appear in some of the fields, so choosing just one separator character and discarding the others would contaminate the data import.
I've had some ideas on how to do this, but they seem kind of hacky (e.g. take three Word8s, test if they're the separator combination, start a new field if they are, recurse further), and I imagine I would be reinventing a wheel anyway. Is there a way to do this without rebuilding the function from scratch?
The documentation of Bytestrings breakSubstring contains a function that does what you are asking for:
tokenise x y = h : if null t then [] else tokenise x (drop (length x) t)
where (h,t) = breakSubstring x y
There are a few functions in bytestring for splitting on subsequences:
breakSubstring :: ByteString -> ByteString -> (ByteString,ByteString)
There's also a
bytestring-csv package, http://hackage.haskell.org/package/bytestring-csv
a split package: http://hackage.haskell.org/package/split for strings though.