Sugar works with map
import sugar
proc map2*[T, R](list: openArray[T]; cb: proc (x: T): R): seq[R] =
for v in list: result.add(cb(v))
result
echo #[1, 2].map2((v) => v*v)
But not with each, seems like it somehow related to void, is there a way to make it work with void too?
import sugar
proc each2*[T](list: openArray[T]; cb: proc (x: T): void): void =
for v in list: cb(v)
#[1, 2].each2((v) => echo v)
I haven't had much luck with sugar.=> personally. This will do what you want, and isn't much more verbose:
proc each2*[T](list: openArray[T]; cb: proc (x: T): void): void =
for v in list: cb(v)
#[1, 2].each2(proc(v: auto) = echo v)
This RFC is proposing proper lambda support, which should give us a succinct way to write anonymous procs that work everywhere.
Related
I need to implement a parallel method, which takes two computation blocks, a and b, and starts each of them in a new thread. The method must return a tuple with the result values of both the computations. It should have the following signature:
def parallel[A, B](a: => A, b: => B): (A, B)
I managed to solve the exercise by using straight Java-like approach. Then I decided to make up a solution with implicit class. Here's it:
object ParallelApp extends App {
implicit class ParallelOps[A](a: => A) {
var result: A = _
def spawn(): Unit = {
val thread = new Thread {
override def run(): Unit = {
result = a
}
}
thread.start()
thread.join()
}
}
def parallel[A, B](a: => A, b: => B): (A, B) = {
a.spawn()
b.spawn()
(a.result, b.result)
}
println(parallel(1 + 2, "a" + "b"))
}
For unknown reason, I receive output (null,null). Could you please point me out where is the problem?
Spoiler alert: It's not complicated. It's funny, like a magic trick (if you consider reading the documentation about Java Memory Model "funny", that is). If you haven't figured it out yet, I would highly recommend to try to figure it out, otherwise it won't be funny. Someone should make a "division-by-zero proves 2 = 4"-riddle out of it.
Consider the following shorter example:
implicit class Foo[A](a: A) {
var result: String = "not initialized"
def computeResult(): Unit = result = "Yay, result!"
}
val a = "a string"
a.computeResult()
println(a.result)
When run, it prints
not initialized
despite the fact that we invoked computeResult() and set result to "Yay, result!". The problem is that the two invocations a.computeResult() and a.result belong to two completely independent instances of Foo. The implicit conversion is performed twice, and the second implicitly created object doesn't know anything about the changes in the first implicitly created object. It has nothing to do with threads or JMM at all.
By the way: your code is not parallel. Calling join right after calling start doesn't bring you anything, your main thread will simply go idle and wait until another thread finishes. At no point will there be two threads that do any useful work concurrently.
EDIT: Fixed a bug pointed out by Andrey Tyukin
One way to solve your problem is to use Scala Futures
Documentation. Tutorial.
Useful Klang Blog.
You'll typically need some combination of these libraries:
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Await, Future}
import scala.util.{Failure, Success}
import scala.concurrent.duration._
an asynchronous example:
def parallelAsync[A,B](a: => A, b: => B): Future[(A,B)] = {
// as per Andrey Tyukin's comments, this line runs
// the two futures sequentially and we do not get
// any benefit from it. I will leave this line here
// so others will not fall in my trap
//for {i <- Future(a); j <- Future(b) } yield (i,j)
Future(a) zip Future(b)
}
parallelAsync(1 + 2, "a" + "b").onComplete {
case Success(x) => println(x)
case Failure(e) => e.printStackTrace()
}
If you must block until both are complete, you can use this:
def parallelSync[A,B](a: => A, b: => B): (A,B) = {
// see comment above
//val f = for { i <- Future(a); j <- Future(b) } yield (i,j)
val tuple = Future(a) zip Future(b)
Await.result(tuple, 5 second)
}
println(parallelSync(3 + 4, "c" + "d"))
When running these little examples, don't forget to sleep a little bit at the end so the program won't end before the results come back
Thread.sleep(3000)
Suppose I have the following nested for loop:
val test = mutableSetOf<Set<Int>>()
for (a in setA) {
for (b in setB) {
if (a.toString().slice(2..3) == b.toString().slice(0..1)) {
test.add(setOf(a,b))
}
}
}
In python, I could do a simple comprehension as
test = {[a,b] for a in setA for b in setB if a.str()[2:3] == b.str[0:1]}
I'm having a helluva time converting this to Kotlin syntax. I know for a single for loop with a conditional, I could use a filter and map to get the desired results (using the idiom: newSet = oldSet.filter{ conditional }.map { it }, but I cannot for the life of me figure out how to do the nesting this way.
This is what IDEA proposes:
for (a in setA)
setB
.filter { a.toString().slice(2..3) == it.toString().slice(0..1) }
.mapTo(test) { setOf(a, it) }
I do not think there is much to do about it. I think their is no native approach that is similar to the Python one, but it already actually is in terms of length very similar because only the functions and their names make it that long.
If we take a look a this hypothetical example:
for (a in setA) setB.f { a.t().s(2..3) == it.t().s(0..1) }.m(test) { setOf(a, it) }
It is not far from the Python example. The Python syntax is just very different.
(functions for that hypothesis)
fun <T> Iterable<T>.f(predicate: (T) -> Boolean) = filter(predicate)
fun String.s(range: IntRange) = slice(range)
fun <T, R, C : MutableCollection<in R>> Iterable<T>.m(destination: C, transform: (T) -> R) = mapTo(destination, transform)
fun Int.t() = toString()
If Kotlin doesn't have it, add it. Here is a cartesian product of the two sets as a sequence:
fun <F,S> Collection<F>.cartesian(other: Collection<S>): Sequence<Pair<F,S>> =
this.asSequence().map { f -> other.asSequence().map { s-> f to s } }.flatten()
Then use that in one of many ways:
// close to your original nested loop version:
setA.cartesian(setB).filter { (a,b) ->
a.toString().slice(2..3) == b.toString().slice(0..1)
}.forEach{ (a,b) -> test.add(setOf(a,b)) }
// or, add the pair instead of a set if that makes sense as alternative
setA.cartesian(setB).filter { (a,b) ->
a.toString().slice(2..3) == b.toString().slice(0..1)
}.forEach{ test2.add(it) }
// or, add the results of the full expression to the set at once
test.addAll(setA.cartesian(setB).filter { (a,b) ->
a.toString().slice(2..3) == b.toString().slice(0..1)
}.map { (a,b) -> setOf(a,b) } )
// or, the same as the last using a pair instead of 2 member set
test2.addAll(setA.cartesian(setB).filter { (a,b) ->
a.toString().slice(2..3) == b.toString().slice(0..1)
})
The above examples use these variables:
val test = mutableSetOf<Set<Int>>()
val test2 = mutableSetOf<Pair<Int,Int>>()
val setA = setOf<Int>()
val setB = setOf<Int>()
My core question is: how can I implement synchronization in a method on the combination of the object instance and the method parameter?
Here are the details of my situation. I'm using the following code to implement memoization, adapted from this answer:
/**
* Memoizes a unary function
* #param f the function to memoize
* #tparam T the argument type
* #tparam R the result type
*/
class Memoized[-T, +R](f: T => R) extends (T => R) {
import scala.collection.mutable
private[this] val cache = mutable.Map.empty[T, R]
def apply(x: T): R = cache.getOrElse(x, {
val y = f(x)
cache += ((x, y))
y
})
}
In my project, I'm memoizing Futures to deduplicate asynchronous API calls. This worked fine when using for...yield to map over the resulting futures, created with the standard ExcecutionContext, but when I upgraded to Scala Async for nicer handling of these futures. However, I realized that the multithreading that library uses allowed multiple threads to enter apply, defeating memoization, because the async blocks all executed in parallel, entering the "orElse" thunk before cache could be updated with a new Future.
To work around this, I put the main apply function in a this.synchronized block:
def apply(x: T): R = this.synchronized {
cache.getOrElse(x, {
val y = f(x)
cache += ((x, y))
y
})
}
This restored the memoized behavior. The drawback is that this will block calls with different params, at least until the Future is created. I'm wondering if there is a way to set up finer grained synchronization on the combination of the Memoized instance and the value of the x parameter to apply. That way, only calls that would be deduplicated will be blocked.
As a side note, I'm not sure this is truly performance critical, because the synchronized block will release once the Future is created and returned (I think?). But if there are any concerns with this that I'm not thinking of, I would also like to know.
Akka actors combined with futures provide a powerful way to wrap over mutable state without blocking. Here is a simple example of how to use an Actor for memoization:
import akka.actor._
import akka.util.Timeout
import akka.pattern.ask
import scala.concurrent._
import scala.concurrent.duration._
class Memoize(system: ActorSystem) {
class CacheActor(f: Any => Future[Any]) extends Actor {
private[this] val cache = scala.collection.mutable.Map.empty[Any, Future[Any]]
def receive = {
case x => sender ! cache.getOrElseUpdate(x, f(x))
}
}
def apply[K, V](f: K => Future[V]): K => Future[V] = {
val fCast = f.asInstanceOf[Any => Future[Any]]
val actorRef = system.actorOf(Props(new CacheActor(fCast)))
implicit val timeout = Timeout(5.seconds)
import system.dispatcher
x => actorRef.ask(x).asInstanceOf[Future[Future[V]]].flatMap(identity)
}
}
We can use it like:
val system = ActorSystem()
val memoize = new Memoize(system)
val f = memoize { x: Int =>
println("Computing for " + x)
scala.concurrent.Future.successful {
Thread.sleep(1000)
x + 1
}
}
import system.dispatcher
f(5).foreach(println)
f(5).foreach(println)
And "Computing for 5" will only print a single time, but "6" will print twice.
There are some scary looking asInstanceOf calls, but it is perfectly type-safe.
I am attempting to create a Scala method that will take one parent group of parentheses, represented as a String, and then map each subgroup of parentheses to a different letter. It should then put these in a map which it returns, so basically I call the following method like this:
val s = "((2((x+3)+6)))"
val map = mapParentheses(s)
Where s could contain any number of sets of parentheses, and the Map returned should contain:
"(x+3)" -> 'a'
"(a+6)" -> 'b'
"(2b)" -> 'c'
"(c)" -> 'd'
So that elsewhere in my program I can recall 'd' and get "(c)" which will become "((2b))" then ((2(a+6))) and finally ((2((x+3)+6))). The string sent to the method mapParentheses will never have unmatched parentheses, or extra chars outside of the main parent parentheses, so the following items will never be sent:
"(fsf)a" because the a is outside the parent parentheses
"(a(aa))(a)" because the (a) is outside the parent parentheses
"((a)" because the parentheses are unmatched
")a(" because the parentheses are unmatched
So I was wondering if anyone knew of an easy (or not easy) way of creating this mapParentheses method.
You can do this pretty easily with Scala's parser combinators. First for the import and some simple data structures:
import scala.collection.mutable.Queue
import scala.util.parsing.combinator._
sealed trait Block {
def text: String
}
case class Stuff(text: String) extends Block
case class Paren(m: List[(String, Char)]) extends Block {
val text = m.head._2.toString
def toMap = m.map { case (k, v) => "(" + k + ")" -> v }.toMap
}
I.e., a block represents a substring of the input that is either some non-parenthetical stuff or a parenthetical.
Now for the parser itself:
class ParenParser(fresh: Queue[Char]) extends RegexParsers {
val stuff: Parser[Stuff] = "[^\\(\\)]+".r ^^ (Stuff(_))
def paren: Parser[Paren] = ("(" ~> insides <~ ")") ^^ {
case (s, m) => Paren((s -> fresh.dequeue) :: m)
}
def insides: Parser[(String, List[(String, Char)])] =
rep1(paren | stuff) ^^ { blocks =>
val s = blocks.flatMap(_.text)(collection.breakOut)
val m = blocks.collect {
case Paren(n) => n
}.foldLeft(List.empty[(String, Char)])(_ ++ _)
(s, m)
}
def parse(input: String) = this.parseAll(paren, input).get.toMap
}
Using get in the last line is very much not ideal, but is justified by your assertion that we can expect well-formed input.
Now we can create a new parser and pass in a mutable queue with some fresh variables:
val parser = new ParenParser(Queue('a', 'b', 'c', 'd', 'e', 'f'))
And now try out your test string:
scala> println(parser parse "((2((x+3)+6)))")
Map((c) -> d, (2b) -> c, (a+6) -> b, (x+3) -> a)
As desired. A more interesting exercise (left to the reader) would be to thread some state through the parser to avoid the mutable queue.
Classic recursive parsing problem. It can be handy to hold the different bits. We'll add a few utility methods to help us out later.
trait Part {
def text: String
override def toString = text
}
class Text(val text: String) extends Part {}
class Parens(val contents: Seq[Part]) extends Part {
val text = "(" + contents.mkString + ")"
def mapText(m: Map[Parens, Char]) = {
val inside = contents.collect{
case p: Parens => m(p).toString
case x => x.toString
}
"(" + inside.mkString + ")"
}
override def equals(a: Any) = a match {
case p: Parens => text == p.text
case _ => false
}
override def hashCode = text.hashCode
}
Now you need to parse into these things:
def str2parens(s: String): (Parens, String) = {
def fail = throw new Exception("Wait, you told me the input would be perfect.")
if (s(0) != '(') fail
def parts(s: String, found: Seq[Part] = Vector.empty): (Seq[Part], String) = {
if (s(0)==')') (found,s)
else if (s(0)=='(') {
val (p,s2) = str2parens(s)
parts(s2, found :+ p)
}
else {
val (tx,s2) = s.span(c => c != '(' && c != ')')
parts(s2, found :+ new Text(tx))
}
}
val (inside, more) = parts(s.tail)
if (more(0)!=')') fail
(new Parens(inside), more.tail)
}
Now we've got the whole thing parsed. So let's find all the bits.
def findParens(p: Parens): Set[Parens] = {
val inside = p.contents.collect{ case q: Parens => findParens(q) }
inside.foldLeft(Set(p)){_ | _}
}
Now we can build the map you want.
def mapParentheses(s: String) = {
val (p,_) = str2parens(s)
val pmap = findParens(p).toSeq.sortBy(_.text.length).zipWithIndex.toMap
val p2c = pmap.mapValues(i => ('a'+i).toChar)
p2c.map{ case(p,c) => (p.mapText(p2c), c) }.toMap
}
Evidence that it works:
scala> val s = "((2((x+3)+6)))"
s: java.lang.String = ((2((x+3)+6)))
scala> val map = mapParentheses(s)
map: scala.collection.immutable.Map[java.lang.String,Char] =
Map((x+3) -> a, (a+6) -> b, (2b) -> c, (c) -> d)
I will leave it as an exercise to the reader to figure out how it works, with the hint that recursion is a really powerful way to parse recursive structures.
def parse(s: String,
c: Char = 'a', out: Map[Char, String] = Map() ): Option[Map[Char, String]] =
"""\([^\(\)]*\)""".r.findFirstIn(s) match {
case Some(m) => parse(s.replace(m, c.toString), (c + 1).toChar , out + (c -> m))
case None if s.length == 1 => Some(out)
case _ => None
}
This outputs an Option containing a Map if it parses, which is better than throwing an exception if it doesn't. I suspect you really wanted a map from Char to the String, so that's what this outputs. c and out are default parameters so you don't need to input them yourself. The regex just means "any number of characters that aren't parens, eclosed in parens" (the paren characters need to be escaped with "\"). findFirstIn finds the first match and returns an Option[String], which we can pattern match on, replacing that string with the relevant character.
val s = "((2((x+3)+6)))"
parse(s) //Some(Map(a -> (x+3), b -> (a+6), c -> (2b), d -> (c)))
parse("(a(aa))(a)") //None
I'm writing up a series of graph-searching algorithms in F# and thought it would be nice to take advantage of parallelization. I wanted to execute several threads in parallel and take the result of the first one to finish. I've got an implementation, but it's not pretty.
Two questions: is there a standard name for this sort of function? Not a Join or a JoinAll, but a JoinFirst? Second, is there a more idiomatic way to do this?
//implementation
let makeAsync (locker:obj) (shared:'a option ref) (f:unit->'a) =
async {
let result = f()
Monitor.Enter locker
shared := Some result
Monitor.Pulse locker
Monitor.Exit locker
}
let firstFinished test work =
let result = ref Option.None
let locker = new obj()
let cancel = new CancellationTokenSource()
work |> List.map (makeAsync locker result) |> List.map (fun a-> Async.StartAsTask(a, TaskCreationOptions.None, cancel.Token)) |> ignore
Monitor.Enter locker
while (result.Value.IsNone || (not <| test result.Value.Value)) do
Monitor.Wait locker |> ignore
Monitor.Exit locker
cancel.Cancel()
match result.Value with
| Some x-> x
| None -> failwith "Don't pass in an empty list"
//end implentation
//testing
let delayReturn (ms:int) value =
fun ()->
Thread.Sleep ms
value
let test () =
let work = [ delayReturn 1000 "First!"; delayReturn 5000 "Second!" ]
let result = firstFinished (fun _->true) work
printfn "%s" result
Would it work to pass the CancellationTokenSource and test to each async and have the first that computes a valid result cancel the others?
let makeAsync (cancel:CancellationTokenSource) test f =
let rec loop() =
async {
if cancel.IsCancellationRequested then
return None
else
let result = f()
if test result then
cancel.Cancel()
return Some result
else return! loop()
}
loop()
let firstFinished test work =
match work with
| [] -> invalidArg "work" "Don't pass in an empty list"
| _ ->
let cancel = new CancellationTokenSource()
work
|> Seq.map (makeAsync cancel test)
|> Seq.toArray
|> Async.Parallel
|> Async.RunSynchronously
|> Array.pick id
This approach makes several improvements: 1) it uses only async (it's not mixed with Task, which is an alternative for doing the same thing--async is more idiomatic in F#); 2) there's no shared state, other than CancellationTokenSource, which was designed for that purpose; 3) the clean function-chaining approach makes it easy to add additional logic/transformations to the pipeline, including trivially enabling/disabling parallelism.
With the Task Parallel Library in .NET 4, this is called WaitAny. For example, the following snippet creates 10 tasks and waits for any of them to complete:
open System.Threading
Array.init 10 (fun _ ->
Tasks.Task.Factory.StartNew(fun () ->
Thread.Sleep 1000))
|> Tasks.Task.WaitAny
In case you are ok to use "Reactive extensions (Rx)" in your project, the joinFirst method can be implemented as:
let joinFirst (f : (unit->'a) list) =
let c = new CancellationTokenSource()
let o = f |> List.map (fun i ->
let j = fun() -> Async.RunSynchronously (async {return i() },-1,c.Token)
Observable.Defer(fun() -> Observable.Start(j))
)
|> Observable.Amb
let r = o.First()
c.Cancel()
r
Example usage:
[20..30] |> List.map (fun i -> fun() -> Thread.Sleep(i*100); printfn "%d" i; i)
|> joinFirst |> printfn "Done %A"
Console.Read() |> ignore
Update:
Using Mailbox processor :
type WorkMessage<'a> =
Done of 'a
| GetFirstDone of AsyncReplyChannel<'a>
let joinFirst (f : (unit->'a) list) =
let c = new CancellationTokenSource()
let m = MailboxProcessor<WorkMessage<'a>>.Start(
fun mbox -> async {
let afterDone a m =
match m with
| GetFirstDone rc ->
rc.Reply(a);
Some(async {return ()})
| _ -> None
let getDone m =
match m with
|Done a ->
c.Cancel()
Some (async {
do! mbox.Scan(afterDone a)
})
|_ -> None
do! mbox.Scan(getDone)
return ()
} )
f
|> List.iter(fun t -> try
Async.RunSynchronously (async {let out = t()
m.Post(Done out)
return ()},-1,c.Token)
with
_ -> ())
m.PostAndReply(fun rc -> GetFirstDone rc)
Unfortunately, there is no built-in operation for this provided by Async, but I'd still use F# asyncs, because they directly support cancellation. When you start a workflow using Async.Start, you can pass it a cancellation token and the workflow will automatically stop if the token is cancelled.
This means that you have to start workflows explicitly (instead of using Async.Parallel), so the synchronizataion must be written by hand. Here is a simple version of Async.Choice method that does that (at the moment, it doesn't handle exceptions):
open System.Threading
type Microsoft.FSharp.Control.Async with
/// Takes several asynchronous workflows and returns
/// the result of the first workflow that successfuly completes
static member Choice(workflows) =
Async.FromContinuations(fun (cont, _, _) ->
let cts = new CancellationTokenSource()
let completed = ref false
let lockObj = new obj()
let synchronized f = lock lockObj f
/// Called when a result is available - the function uses locks
/// to make sure that it calls the continuation only once
let completeOnce res =
let run =
synchronized(fun () ->
if completed.Value then false
else completed := true; true)
if run then cont res
/// Workflow that will be started for each argument - run the
/// operation, cancel pending workflows and then return result
let runWorkflow workflow = async {
let! res = workflow
cts.Cancel()
completeOnce res }
// Start all workflows using cancellation token
for work in workflows do
Async.Start(runWorkflow work, cts.Token) )
Once we write this operation (which is a bit complex, but has to be written only once), solving the problem is quite easy. You can write your operations as async workflows and they'll be cancelled automatically when the first one completes:
let delayReturn n s = async {
do! Async.Sleep(n)
printfn "returning %s" s
return s }
Async.Choice [ delayReturn 1000 "First!"; delayReturn 5000 "Second!" ]
|> Async.RunSynchronously
When you run this, it will print only "returning First!" because the second workflow will be cancelled.