Can't understand multithreading in clojure. Can't find examples of REAL multithreading. Most samples with atoms, refs, vars are singlethreaded. So, I have a quest. Two threads gaining access to one field, each thread can change it. I use atom for this purpose, so the Code is:
(do
(def field (atom "v0"))
(defn f1 []
(dotimes [i 100000]
(if (= i 9999)
(reset! field "v1"))))
(defn f2 []
(dotimes [i 100000]
(if (= i 777)
(reset! field "v2"))))
(do
(deref (future (Thread/sleep 10) (f1))
0 f2)
(prn #field)))
But nothing, the value of field is "v0". How to make normal twothreaded example with cycles in each thread and with access to variable???
watch the docs of deref:
clojure.core/deref
([ref] [ref timeout-ms timeout-val])
returns the in-transaction-value of ref, else returns the
most-recently-committed value of ref. When applied to a var, agent
or atom, returns its current state. When applied to a delay, forces
it if not already forced. When applied to a future, will block if
computation not complete. When applied to a promise, will block
until a value is delivered. The variant taking a timeout can be
used for blocking references (futures and promises), and will return
timeout-val if the timeout (in milliseconds) is reached before a
value is available. See also - realized?.
so your timeout is 0, that means it will return default value
which is f2 - a function value (not a function call), which is not being called obviously, so no reset! ever happens.
if you want "v1" you should deref like:
(deref (future (Thread/sleep 10) (f1)) 100 (f2))
if you want "v2":
(deref (future (Thread/sleep 10) (f1)) 0 (f2))
Related
I have an immutable datastructure that is a functional hashmap (see fash.scm) that is shared among several threads.
Imagine a thread wants to change the global hashmap to a new version. Do I need a lock to change the value? If that's is the case, I assume I also need to lock the value to read it, isn't it?
It seems to me it boils down to whether setting a value in Scheme is an atomic operation or not. According to this answer for C language, you must
acquire a lock for both read and write access of pointer.
If it matters I am using guile 2.2.3 and bigloo 4.3.
It all depends on what you want to do. In general, if the value can be guaranteed to be read (e.g. always a number), then it's okay not to lock when the value is read. For example,
(import (rnrs) (srfi :18))
(define count 0)
(define t
(thread-start!
(make-thread
(lambda ()
(let loop ()
(display count) (newline)
(thread-sleep! 0.1)
(loop))))))
(do ((i 0 (+ i 1))) ((= i 10))
(set! count (+ count 1))
(thread-sleep! 0.1))
This is pretty much safe to read. However if the value is, say a vector of length 1, then you may want to lock if the other threads may change the value to #f or a vector of length 0. For example:
(import (rnrs) (srfi :18))
(define count (vector 1))
(define t
(thread-start!
(make-thread
(lambda ()
(let loop ()
(display (vector-ref count 0)) (newline)
(thread-sleep! 0.1)
(loop))))))
(do ((i 0 (+ i 1))) ((= i 10))
(vector-set! count 0 (+ (vector-ref count 0) i))
(thread-sleep! 0.1))
(set! count #f) ;; if this happens, the you may want to lock on reader thread
I haven't check how fash is implemented, but as long as entries are not updated to unexpected values, I would think it's okay not to lock.
I'm trying to update a nested counter in an atom (a map) from multiple threads, but getting unpredictable results.
(def a (atom {:id {:counter 0}}))
(defn upvote [id]
(swap! a assoc-in [(keyword id) :counter] (inc (get-in #a [(keyword id) :counter])))
)
(dotimes [i 10] (.start (Thread. (fn [] (upvote "id")))))
(Thread/sleep 12000)
(prn #a)
I'm new to Clojure so very possible I'm doing something wrong, but can't figure out what. It's printing a counter value with results varying from 4-10, different each time.
I want to atomically update the counter value and hoped that this approach would always give me a counter value of 10. That it would just retry upon failure and eventually get to 10.
It's for an up-vote function that can get triggered concurrently.
Can you see what I'm doing wrong here?
You are updating the atom non-atomically in your code. You first get its value by #a, and then apply it using the swap function. The value may change in between.
The atomic way to update the value is to use a pure function within swap, without referring to the previous atom value via #:
(defn upvote [id]
(swap! a update-in [(keyword id) :counter] inc))
I have a Clojure code which runs a few threads in parallel. They all share an atom: (def counter (atom 0)) which is incremented by each thread. Every 10 minute, I'd like to perform several actions using the value of the atom and then reset it back to 0 - for example:
(defn publish-val []
(let [c #counter]
(email c)
(statsd c)
(print-log c)
(reset! counter 0)))
It is important that the value of counter will not change from the moment it is dereferenced to the moment it is reset - meaning all threads should be blocked when trying to change the atom's value while publish-val is executed. How do I do this?
Unless you've greatly simplified the problem for your example, it looks like swap!-ing out the current counter value with zero would be sufficient here:
(defn publish-val []
(with-local-vars [c nil]
(swap! counter
(fn [x] (var-set c x) 0))
(email #c)
(statsd #c)
(print-log #c)))
So you just save your old counter value in a local variable, atomically swap it with zero, and then do whatever bookkeeping needed to do with the old value—all without stalling any of the other threads longer than it takes to swap!
Use an agent.
See the section Using agents to serialise access to non-threadsafe resources about using them to print to console
Suppose that channel chan has the values "1" and "2" on queue.
Goal: Make a function which takes chan and returns the vector [1 2]. Note that I am totally fine if this function has to block for some time before its value is returned.
Attempt:
(defn chan->vector
[chan]
(let [a (atom true) v []]
(while (not-nil? #a)
(go
(reset! a (<! chan))
(into v #a)
(reset! a (<! chan))
)
) v
)
)
Result: My REPL freezes and eventually spits out a huge error. I have come to realize that this is because the (go ...) block is asynchronous, and so immediately returns. Thus the atom іn my (while ...) loop is never given a chance to be set to nil and the loop can never terminate.
So how do I accomplish the desired result? In case it's relevant, I'm using ClojureScript and targetting nodejs.
you should use alts! from core.async to fulfill this task
(https://clojure.github.io/core.async/#clojure.core.async/alts!):
(def x (chan 10))
(go (>! x 1)
(>! x 2)
(>! x 3))
(defn read-all [from-chan]
(<!! (go-loop [res []]
(let [[v _] (alts! [from-chan] :default :complete)]
(if (= v :complete)
res
(recur (conj res v)))))))
(read-all x)
;; output: [1 2 3]
(read-all x)
;; output: []
(go (>! x 10)
(>! x 20)
(>! x 30)
(>! x 40))
(read-all x)
;; output: [10 20 30 40]
inside the go-loop this (a/alts! [from-chan] :default :complete) tries to read any value from channel, and in case there are no value available at once it emits the default value, so you will see you should break the loop and return accumulated values.
update: as the blocking read (<!!) is absent in cljs, you can rewrite it the following way:
(defn read-all [from-chan]
(go-loop [res []]
(let [[v _] (alts! [from-chan] :default :complete)]
(if (= v :complete)
res
(recur (conj res v)))))))
so it will return the channel, and then just read one value from there:
(go (let [res (<! (read-all x))]
(println res)
;; do something else
))
You can use clojure.core.async/reduce:
;; demo setup
(def ch (async/chan 2))
(async/>!! ch :foo)
(async/>!! ch :bar)
;; background thread to print reduction result
(async/thread
(prn (async/<!! (async/reduce conj [] ch))))
;; closing the channel…
(async/close! ch)
;; …terminates the reduction and the result gets printed out:
;; [:foo :bar]
clojure.core.async/reduce returns a channel that will produce a value if and when the original channel closes. Internally it uses a go block and will release control in between taking elements from the original channel.
If you want to produce a value after a certain amount of time passes whether or not the original channel closes, you can either wrap the original channel in a pass-through channel that will close itself after a timeout passes or you can use a custom approach to the reduction step (perhaps the approach suggested by #leetwinski).
Use into
Returns a channel containing the single (collection) result of the
items taken from the channel conjoined to the supplied collection. ch
must close before into produces a result.
Something like this should work (it should print the events from events-chan given events-chan closes when it is done publishing events):
(go
(println (<! (into [] events-chan))))
The source channel needs to end (close), otherwise you can't put all events into a collection.
Edit:
Re-read your question, and it is not very clear what you want to accomplish. Whatever you want to do, chan->vector needs to return a channel so that whoever calls it can wait for the result. In fact, chan->vector is exactly into:
; chan->vector ch:Chan<Event> -> Chan<Vector[Event]>
(defn chan->vector [ch]
(into [] ch))
(go
(let [events (<! (chan->vector events-chan))]
(println events) ; Do whatever with the events vector
))
As I mentioned above, if the events chan never closes, then you have to do more thinking about how to consume the events. There is no magic solution. Do you want to batch the events by time intervals? By number of events? By a combination of those?
In summary, as mentioned above, chan->vector is into.
While possible in Clojure and many other languages, what you want to do is not possible in ClojureScript.
You want a function that blocks while listening to a channel. However, ClojureScript's version of core.async doesn't include the blocking operators. Why? Because ClojureScript doesn't block.
I couldn't find a reliable source to back that last sentence. There seems to be a lot of confusion around this topic on the web. However, I'm pretty sure of what I'm saying because ClojureScript ultimately becomes JavaScript, and that's how JavaScript works.
Indeed, JavaScript never blocks, neither on the browser nor in Node.js. Why? As far as I understand, it uses a single thread, so if it were to block, the user would be unable to do anything in the browser.
So it's impossible to do what you want. This is by design, because it could have disastrous UX effects. ClojureScript channels are like JavaScript events; in the same way you don't want an event listener to block the user interface while waiting for an event to happen, you also shouldn't want a channel to block while waiting for new values.
Instead, try using a callback function that gets called whenever a new value is delivered.
I'm implementing a mechanism for a thread to have a queue that contains messages. The queue is built using LinkedBlockingQueue from java.util.concurrent. What I want to achieve is something like the following.
Thread with mailbox:
defn work:
* do some stuff
* Get the head of the queue (a message):
- if it is "hello":
<do some stuff>
<recur work fn>
- if it is "bye":
<do some stuff>
- if it is none of the above, add the message to the back of queue
and restart from "Get the head of the queue"
* <reaching this point implies terminating the thread>
My first idea that I tried to implement was using a loop wrapped around the * Get the head of the queue, use a conditional to check the message and add it to the queue in an :else branch if it did not match any of the clauses. The downside of this was that calling recur in any of the bodies of the clauses of the cond would always recur the loop, while using recur (e.g., like in the hello case) means recur the function (i.e., work). So that was not an option. An other downside would be that, in case it would take a long time for such message to arrive the thread would indefinitely spin and eat resources.
The next idea I had (but have not yet implemented) is using a future. The scheme would be as follows.
* Get all the matches I have to match (i.e., "hello" and "bye")
* Start a future and pass it the list of messages:
* While the queue does not contain any of the messages
recur
* when found, return the first element that matches.
* Wait for the future to deliver.
* if it is "hello":
<do some stuff>
<recur work fn>
if it is "bye":
<do some stuff>
When doing it this way I get almost what I want:
Receiving either "hello" or "bye" blocks until I have either one.
I can make an indefinite number of clauses to match the message
I have extracted the looping behaviour into a future that blocks, which
has the nice side-effect that each time I evaluate my cond
I'm sure I have a matching message and don't have to worry about retrying.
One thing I really would like, but can't imagine how to achieve, is that the future in this case does not spin. As it stands it would keep eating up precious CPU resources traversing the queue indefinitely, while it might be perfectly normal to never receive one of the messages it is looking for.
Perhaps it would make sense to abandon the LinkedBlockedQueue and trade it in for a data structure that has a method, say, getEither(List<E> oneOfThese) that blocks until one of these elements in available.
An other thought I had, which is a way I could possibly do it in Java, is having the aforementioned getEither() operation on the queue that calls wait() if none of the elements are in the queue. When an other thread puts a message in the queue I can call notify() so that each thread will check the queue against his list of wanted messages.
Example
The code below works fine. However, it has the spinning problem. It's basicly a very elementary example of what I'm trying to achieve.
(def queue (ref '()))
(defn contains-element [elements collection]
(some (zipmap elements (repeat true)) collection))
(defn has-element
[col e]
(some #(= e %) col))
(defn find-first
[f coll]
(first (filter f coll)))
; This function is blocking, which is what I want.
; However, it spins and thus used a LOT of cpu,
; whit is *not* what I want..
(defn get-either
[getthese queue]
(dosync
(let [match (first (filter #(has-element getthese %) #queue))
newlist (filter #(not= match %) #queue)]
(if (not (nil? match))
(do (ref-set queue newlist)
match)
(Thread/sleep 500)
(recur)))))
(defn somethread
[iwantthese]
(let [element (get-either iwantthese queue)
wanted (filter #(not= % element) iwantthese)]
(println (str "I got " element))
(Thread/sleep 500)
(recur wanted)))
(defn test
[]
(.start (Thread. (fn [] (somethread '(3 4 5)))))
(dosync (alter queue #(cons 1 %)))
(println "Main: added 1")
(Thread/sleep 1000)
(dosync (alter queue #(cons 2 %)))
(println "Main: added 2")
(Thread/sleep 1000)
(dosync (alter queue #(cons 3 %)))
(println "Main: added 3")
(Thread/sleep 1000)
(dosync (alter queue #(cons 4 %)))
(println "Main: added 4")
(Thread/sleep 1000)
(dosync (alter queue #(cons 5 %)))
(println "Main: added 5")
)
Any tips?
(In case anyone noticed, yes, this is like actors and the purpose is an implementation in Clojure for academic purposes)
You need 2 queues instead of one: incoming queue and a "dead-letter" queue.
A "thread" should read from the incoming queue in a blocking way ( LinkedBlockingQueue.take(), core.async/<! or using agents).
If message doesn't match any clause:
Put message on end of dead queue
Go to 1.
If message matches a clause:
Run the clause work
For each message in the dead queue, match against clauses, removing the ones that are matched.
go to 1.
See below for the two implementations.
Agents
Agents are quite similar to actors, the "only" difference is that you send data/messages to actors but you send functions to agents. A possible implementation would be:
(defn create-actor [behaviour]
(agent {:dead-queue []
:behaviour behaviour}))
dead-queue will contain messages that didn't match any of the clauses. This is basically your "end of the queue".
behaviour should be some map/vector of match-fn to fn to run. In my particular implementation, I have chosen a map, where keys are the element to match and values are the fn to run when the new item matches:
(def actor (create-actor {3 println
4 (partial println "Got a ")
5 #(println "Got a " %)}))
You will probably require a more sophisticated behaviour data structure. The only thing important is to know if the element was processed or not, so you know if the element has to go to the dead queue or not.
To send messages to the actor:
(defn push [actor message]
(send actor
(fn [state new-message]
(if-let [f (get-in state [:behaviour new-message])]
(do
(f new-message)
state)
(update-in state [:dead-queue] conj new-message)))
message))
So if there is a match on the behaviour, the message is processed immediately. If not, it is stored in the dead queue. You could in try to match/process all the messages in the dead queue after processing the new message if you expected that the behaviours are not pure functions. In this example implementation this is not possible.
We could change the behaviour of the actor to give the messages on the dead queue a chance to be processed:
(defn change-behaviour [actor behaviour]
(send actor
(fn [state new-behaviour]
(let [to-process (filter new-behaviour (:dead-queue state))
new-dead-queue (vec (remove (set to-process) (:dead-queue state)))]
(doseq [old-message to-process
:let [f (get new-behaviour old-message)]]
(f old-message))
{:behaviour new-behaviour
:dead-queue new-dead-queue}))
conds))
And an example of using it:
(push actor 4)
(push actor 18)
(push actor 1)
(push actor 18)
(push actor 5)
(change-behaviour actor {18 (partial println "There was an")})
And the same solution based on core.async:
(defn create-actor [behaviour]
(let [queue (async/chan)]
(async/go-loop [dead-queue []
behaviour behaviour]
(let [[type val] (async/<! queue)]
(if (= type :data)
(if-let [f (get behaviour val)]
(do
(f val)
(recur dead-queue behaviour))
(recur (conj dead-queue val) behaviour))
(let [to-process (filter val dead-queue)
new-dead-queue (vec (remove (set to-process) dead-queue))]
(doseq [old-msg to-process
:let [f (get val old-msg)]]
(f old-msg))
(recur new-dead-queue val)))))
queue))
(defn push [actor message]
(async/go
(async/>! actor [:data message])))
(defn change-behaviour [actor behaviour]
(async/go
(async/>! actor [:behaviour behaviour])))
Have you considered using core.async? It provides what you need in a lightweight way.