I talked about this a bit on IRC's #clojure channel today but would like to go more in detail here. Basically, in order to better understand atoms, swap!, deref and Clojure concurrency as a whole, I'd like to try to write a function which not only returns the value that was swapped-in using swap!, but also the value that was swapped out.
(def foo (atom 42))
.
.
.
((fn [a]
(do
(println "swapped out: " #a)
(println "swapped in: "(swap! a rand-int)))) foo)
may print:
swapped out: 42
swapped in: 14
However if another thread does swap! the same atom between the #a deref and the call to swap! then I may be swapping out a value that is not 42.
How can I write a function which gives back correctly both values (the swapped out and the swapped in)?
I don't care about the various values that the atom does change to: all I want to know is what was the value swapped out.
Can this be written using code that is guaranteed not to deadlock and if so why?
Clojure's swap! is just a spinning compare-and-set. You can define an alternate version that returns whatever you like:
(defn alternate-swap [atom f & args]
(loop []
(let [old #atom
new (apply f old args)]
(if (compare-and-set! atom old new)
[old new] ; return value
(recur)))))
Atoms are un-coordinated so it seems likely that any attempt to do this outside of the swapping function it's self will likely fail. You could write a function that you call instead of swap! which constructs a function that saves the existing value before applying the real function, and then pass this constructed function to swap!.
user> (def foo (atom []))
#'user/foo
user> (defn save-n-swap! [a f & args]
(swap! a (fn [old-val]
(let [new-val (apply f (cons old-val args))]
(println "swapped out: " old-val "\n" "swapped in: " new-val)
new-val))))
#'user/save-n-swap!
user> (save-n-swap! foo conj 4)
swapped out: []
swapped in: [4]
[4]
user> (save-n-swap! foo conj 4)
swapped out: [4]
swapped in: [4 4]
[4 4]
This example prints it, It would also make sense to push them to a changelog stored in another atom
If you want the return value, Stuart answer is the correct one, but if you are just going to do a bunch of println to understand how atoms/refs work, I would recommend to add a watch to the atom/ref http://clojuredocs.org/clojure_core/1.2.0/clojure.core/add-watch
(add-watch your-atom :debug (fn [_ _ old new] (println "out" old "new" new)))
You could use a macro like:
(defmacro swap!-> [atom & args]
`(let [old-val# (atom nil)
new-val# (swap! ~atom #(do
(swap! old-val# (constantly %))
(-> % ~args)))]
{:old #old-val# :new new-val#}))
(def data (atom {}))
(swap!-> data assoc :a 3001)
=> {:new {:a 3001} :old {}}
Refer to swap-vals! available since 1.9: https://clojuredocs.org/clojure.core/swap-vals%21
You could rely on a promise to store the current value inside the swap! operation. Then you return the new and old value in a vector, as follows:
(defn- swap-and-return-old-value!
[^clojure.lang.IAtom atom f & args]
(let [old-value-promise (promise)
new-value (swap! atom
(fn [old-value]
(deliver old-value-promise old-value)
(apply f old-value args)))]
[new-value #old-value-promise]))
Related
I'm trying to return a string value when I read this CSV file that contains cities and city attributes. Here is what I have so far:
(defn city [name]
(with-open [rdr (reader)]
(doseq [line (drop 1 (line-seq rdr))]
(def x2 line)
(def y (string/split x2 #","))
(if (= name (y 0))
(println line)
))))
(city "Toronto")
=> Toronto,43.666667,-79.416667,Canada,CA,Ontario,admin,5213000,3934421
I can get it to print out the row, but how would I go about getting the function to return the row instead, if that makes sense?
With how you have the code setup currently, you can't. doseq is meant to carry out side effects; it doesn't return anything. Rarely do you ever want to use doseq, and rarely should you ever use def inside of function definitions.
You want to find the first line where (= name (y 0)) is true. There's a few ways of approaching that. A basic way would be using loop and just stopping it once you find the line. I think using map or for to loop over the line-seq, then grabbing the first result would work out well here though:
(defn city [name]
(with-open [rdr (reader)]
(first
(for [line (drop 1 (line-seq rdr)) ; Same syntax here as with doseq
:let [y (string/split line #",")] ; Use let instead of def for local definitions
:when (= name (y 0))] ; Only add to the list ":when (= name (y 0))"
line))))
for is like Python's generator expression (if you're familiar with Python). It is not like a normal imperative for loop like in Java. The for will return a list of lines for which (= name (y 0)) was true. Because presumably there's only one such valid line in the file though, we only want one result, so we pass the list to first to get the first valid line found.
And note that for is lazy. This does not iterate the entire file before passing off to first. first requests the first element before for has even iterated, and no more iteration is done once a matching line is found.
The println function is meant for side-effects, and always returns nil. Adjust your function to return line as the last item after the if:
(if (= name (y 0))
line)
If you haven't seen it yet, look at
Brave Clojure (free & book)
Getting Clojure (book)
Clojure Cheatsheet
Here is a better organized version of the code. Your project dependencies will need to look like:
:dependencies [
[org.clojure/clojure "1.10.1"]
[prismatic/schema "1.1.12"]
[tupelo "0.9.168"]
]
and the code can then look like:
(ns tst.demo.core
(:use tupelo.core tupelo.test)
(:require
[clojure.string :as str]
[clojure.java.io :as io]
[tupelo.string :as ts]))
(def city-data
" city,lat,lon,country,ccode,province,unk1,unk2,unk3
Toronto,43.666667,-79.416667,Canada,CA,Ontario,admin,5213000,3934421
Chicago,40.666667,-99.416667,USA,US,dummy,admin,5213000,3934421
")
(defn city->fields [city-str]
(str/split city-str #","))
(defn city [name]
(with-open [rdr (io/reader (ts/string->stream city-data))]
(let [lines (mapv str/trim (line-seq rdr))
hdrs-line (first lines)
city-lines (rest lines)
cities-fields (mapv city->fields city-lines)
city-match (first (filterv #(= name (first %)) cities-fields))]
; debug printouts
(spyx hdrs-line)
(spyx-pretty city-lines)
(spyx city-match)
city-match))) ; <= return value
(dotest
(println "Result: " (city "Toronto"))
)
with result:
-------------------------------
Clojure 1.10.1 Java 13
-------------------------------
Testing tst.demo.core
hdrs-line => "city,lat,lon,country,ccode,province,unk1,unk2,unk3"
city-lines =>
("Toronto,43.666667,-79.416667,Canada,CA,Ontario,admin,5213000,3934421"
"Chicago,40.666667,-99.416667,USA,US,dummy,admin,5213000,3934421"
"")
city-match => ["Toronto" "43.666667" "-79.416667" "Canada" "CA" "Ontario" "admin" "5213000" "3934421"]
Result: [Toronto 43.666667 -79.416667 Canada CA Ontario admin 5213000 3934421]
I'm trying to update a nested counter in an atom (a map) from multiple threads, but getting unpredictable results.
(def a (atom {:id {:counter 0}}))
(defn upvote [id]
(swap! a assoc-in [(keyword id) :counter] (inc (get-in #a [(keyword id) :counter])))
)
(dotimes [i 10] (.start (Thread. (fn [] (upvote "id")))))
(Thread/sleep 12000)
(prn #a)
I'm new to Clojure so very possible I'm doing something wrong, but can't figure out what. It's printing a counter value with results varying from 4-10, different each time.
I want to atomically update the counter value and hoped that this approach would always give me a counter value of 10. That it would just retry upon failure and eventually get to 10.
It's for an up-vote function that can get triggered concurrently.
Can you see what I'm doing wrong here?
You are updating the atom non-atomically in your code. You first get its value by #a, and then apply it using the swap function. The value may change in between.
The atomic way to update the value is to use a pure function within swap, without referring to the previous atom value via #:
(defn upvote [id]
(swap! a update-in [(keyword id) :counter] inc))
Suppose that channel chan has the values "1" and "2" on queue.
Goal: Make a function which takes chan and returns the vector [1 2]. Note that I am totally fine if this function has to block for some time before its value is returned.
Attempt:
(defn chan->vector
[chan]
(let [a (atom true) v []]
(while (not-nil? #a)
(go
(reset! a (<! chan))
(into v #a)
(reset! a (<! chan))
)
) v
)
)
Result: My REPL freezes and eventually spits out a huge error. I have come to realize that this is because the (go ...) block is asynchronous, and so immediately returns. Thus the atom іn my (while ...) loop is never given a chance to be set to nil and the loop can never terminate.
So how do I accomplish the desired result? In case it's relevant, I'm using ClojureScript and targetting nodejs.
you should use alts! from core.async to fulfill this task
(https://clojure.github.io/core.async/#clojure.core.async/alts!):
(def x (chan 10))
(go (>! x 1)
(>! x 2)
(>! x 3))
(defn read-all [from-chan]
(<!! (go-loop [res []]
(let [[v _] (alts! [from-chan] :default :complete)]
(if (= v :complete)
res
(recur (conj res v)))))))
(read-all x)
;; output: [1 2 3]
(read-all x)
;; output: []
(go (>! x 10)
(>! x 20)
(>! x 30)
(>! x 40))
(read-all x)
;; output: [10 20 30 40]
inside the go-loop this (a/alts! [from-chan] :default :complete) tries to read any value from channel, and in case there are no value available at once it emits the default value, so you will see you should break the loop and return accumulated values.
update: as the blocking read (<!!) is absent in cljs, you can rewrite it the following way:
(defn read-all [from-chan]
(go-loop [res []]
(let [[v _] (alts! [from-chan] :default :complete)]
(if (= v :complete)
res
(recur (conj res v)))))))
so it will return the channel, and then just read one value from there:
(go (let [res (<! (read-all x))]
(println res)
;; do something else
))
You can use clojure.core.async/reduce:
;; demo setup
(def ch (async/chan 2))
(async/>!! ch :foo)
(async/>!! ch :bar)
;; background thread to print reduction result
(async/thread
(prn (async/<!! (async/reduce conj [] ch))))
;; closing the channel…
(async/close! ch)
;; …terminates the reduction and the result gets printed out:
;; [:foo :bar]
clojure.core.async/reduce returns a channel that will produce a value if and when the original channel closes. Internally it uses a go block and will release control in between taking elements from the original channel.
If you want to produce a value after a certain amount of time passes whether or not the original channel closes, you can either wrap the original channel in a pass-through channel that will close itself after a timeout passes or you can use a custom approach to the reduction step (perhaps the approach suggested by #leetwinski).
Use into
Returns a channel containing the single (collection) result of the
items taken from the channel conjoined to the supplied collection. ch
must close before into produces a result.
Something like this should work (it should print the events from events-chan given events-chan closes when it is done publishing events):
(go
(println (<! (into [] events-chan))))
The source channel needs to end (close), otherwise you can't put all events into a collection.
Edit:
Re-read your question, and it is not very clear what you want to accomplish. Whatever you want to do, chan->vector needs to return a channel so that whoever calls it can wait for the result. In fact, chan->vector is exactly into:
; chan->vector ch:Chan<Event> -> Chan<Vector[Event]>
(defn chan->vector [ch]
(into [] ch))
(go
(let [events (<! (chan->vector events-chan))]
(println events) ; Do whatever with the events vector
))
As I mentioned above, if the events chan never closes, then you have to do more thinking about how to consume the events. There is no magic solution. Do you want to batch the events by time intervals? By number of events? By a combination of those?
In summary, as mentioned above, chan->vector is into.
While possible in Clojure and many other languages, what you want to do is not possible in ClojureScript.
You want a function that blocks while listening to a channel. However, ClojureScript's version of core.async doesn't include the blocking operators. Why? Because ClojureScript doesn't block.
I couldn't find a reliable source to back that last sentence. There seems to be a lot of confusion around this topic on the web. However, I'm pretty sure of what I'm saying because ClojureScript ultimately becomes JavaScript, and that's how JavaScript works.
Indeed, JavaScript never blocks, neither on the browser nor in Node.js. Why? As far as I understand, it uses a single thread, so if it were to block, the user would be unable to do anything in the browser.
So it's impossible to do what you want. This is by design, because it could have disastrous UX effects. ClojureScript channels are like JavaScript events; in the same way you don't want an event listener to block the user interface while waiting for an event to happen, you also shouldn't want a channel to block while waiting for new values.
Instead, try using a callback function that gets called whenever a new value is delivered.
Can't understand multithreading in clojure. Can't find examples of REAL multithreading. Most samples with atoms, refs, vars are singlethreaded. So, I have a quest. Two threads gaining access to one field, each thread can change it. I use atom for this purpose, so the Code is:
(do
(def field (atom "v0"))
(defn f1 []
(dotimes [i 100000]
(if (= i 9999)
(reset! field "v1"))))
(defn f2 []
(dotimes [i 100000]
(if (= i 777)
(reset! field "v2"))))
(do
(deref (future (Thread/sleep 10) (f1))
0 f2)
(prn #field)))
But nothing, the value of field is "v0". How to make normal twothreaded example with cycles in each thread and with access to variable???
watch the docs of deref:
clojure.core/deref
([ref] [ref timeout-ms timeout-val])
returns the in-transaction-value of ref, else returns the
most-recently-committed value of ref. When applied to a var, agent
or atom, returns its current state. When applied to a delay, forces
it if not already forced. When applied to a future, will block if
computation not complete. When applied to a promise, will block
until a value is delivered. The variant taking a timeout can be
used for blocking references (futures and promises), and will return
timeout-val if the timeout (in milliseconds) is reached before a
value is available. See also - realized?.
so your timeout is 0, that means it will return default value
which is f2 - a function value (not a function call), which is not being called obviously, so no reset! ever happens.
if you want "v1" you should deref like:
(deref (future (Thread/sleep 10) (f1)) 100 (f2))
if you want "v2":
(deref (future (Thread/sleep 10) (f1)) 0 (f2))
I have a string containing a valid Clojure form. I want to replace a part of it, just like with assoc-in, but processing the whole string as tokens.
=> (assoc-in [:a [:b :c]] [1 0] :new)
[:a [:new :c]]
=> (assoc-in [:a
[:b,, :c]] [1 0] :new)
[:a [:new :c]]
=> (string-assoc-in "[:a
[:b,, :c]]" [1 0] ":new")
"[:a
[:new,, :c]]"
I want to write string-assoc-in. Note that its first and last arguments are strings, and it keeps the line break and the commas. Is it doable in Clojure? The closest thing I found is read which calls clojure.lang.LispReader, but I don't know how works.
I want to use it to read a Clojure source file and display it with some modifications, keeping the structure of the file.
Or another option would be to use ANTLR to parse the Clojure code into an AST, then transform the AST, and export back out to a string.
I think this should work, be entirely general and not require its own reader / parser:
(defn is-clojure-whitespace? [c]
(or (Character/isSpace c)
(= \, c)))
(defn whitespace-split
"Returns a map of true -> (maximal contiguous substrings of s
consisting of Clojure whitespace), false -> (as above, non-whitespace),
:starts-on-whitespace? -> (whether s starts on whitespace)."
[s]
(if (empty? s)
{}
(assoc (group-by (comp is-clojure-whitespace? first)
(map (partial apply str)
(partition-by is-clojure-whitespace? s)))
:starts-on-whitespace?
(if (is-clojure-whitespace? (first s)) true false))))
(defn string-assoc-in [s coords subst]
(let [{space-blocks true
starts-on-whitespace? :starts-on-whitespace?}
(whitespace-split s)
s-obj (assoc-in (binding [*read-eval* false] (read-string s))
coords
(binding [*read-eval* false] (read-string subst)))
{non-space-blocks false}
(whitespace-split (pr-str s-obj))]
(apply str
(if starts-on-whitespace?
(interleave space-blocks (concat non-space-blocks [nil]))
(interleave non-space-blocks (concat space-blocks [nil]))))))
Example:
user> (string-assoc-in "[:a [:b,, :c]]" [1 0] ":new")
"[:a [:new,, :c]]"
Update: Ouch, caught a bug:
user> (string-assoc-in "[:a [:b,, :c\n]]" [1 0] ":new")
"[:a [:new,, :c]]\n"
I'd love it if it didn't matter, but I guess I'll have to try and do something about it... sigh
You could do this with a combination of (read-string) and some string manipulation:
(defn string-assoc-in
[a b c]
(.replaceAll
(str
(assoc-in (read-string (.replaceAll a ",," ",_,")) b (read-string c)))
" _ " ",, "))
user> (string-assoc-in "[:a [:b,, :c]]" [1 0] ":new")
"[:a [:new,, :c]]"
Note that we require a reserved placeholder character (in this case, _) which you wouldn't want in your keywords. The trick is to get those ,, out of the way when the reader is crunching on the vector string, then put them back.
This sample doesn't address the newlines, but I think you could handle those in the same way.
I'm assuming you don't want to actually read in a form and evaluate it? fnparse has a Clojure parser (written in Clojure using fnparse). You might be able to use that to get you from string to form, then manipulate, then put it back out to a string?