I use some Java library that makes non-async get and post requests. I used to wrap such requests to futures and it solves for me the "waiting problem" (I mean waiting for the response)
(defn unchangeable-lib-request [n]
(Thread/sleep 1000)
n)
(defn process [n]
(let [res (atom [])]
(dotimes [i n]
(future (swap! res conj (unchangeable-lib-request i))))
(loop []
(if (> n (count #res))
(recur)
#res))))
(time (process 9))
;; "Elapsed time: 1000.639079 msecs"
;; => [8 7 5 6 4 3 2 1 0]
But I need to create hundreds of requests and this creates performance problems. I found out about core.async and go blocks. But if I will use go-blocks with this library, it will not solve the "waiting problem"
(defn unchangeable-lib-request [n]
(Thread/sleep 1000)
n)
(defn process [n]
(let [c (async/chan 10)]
(dotimes [i n]
(async/go
(async/>! c (unchangeable-lib-request i))))
(loop [result []]
(if (> n (count result))
(recur (conj result (async/<!! c)))
result))))
(time (process 9))
;; "Elapsed time: 2001.770183 msecs"
;; => [0 4 1 6 7 2 5 3 8]
Go blocks can handle just 8 requests simultaneously. Is there a possibility to write some async-wrapper that will park go-block and provide ability to make 100s of requests asynchronously without blocking each other?
(defn process [n]
(let [c (async/chan 10)]
(dotimes [i n]
(async/go
(async/>! c (magic-async-parking-wrapper
(unchangeable-lib-request i))))
(loop [result []]
(if (> n (count result))
(recur (conj result (async/<!! c)))
result))))
(time (process 9))
;; "Elapsed time: 1003.2563 msecs"
I know about async/thread but it seems that this is the same as (future ...).
Is it possible?
I'd suggest:
Use futures to create the threads, and have them put the results back onto a core async channel from outside of any go block using put!, something like: (future (put! chan (worker-function)))
Then use a go block to wait on that (single) channel, put in the results as you get them.
This is where you use clojure.core.async/pipeline-blocking
(require '[clojure.core.async :as a :refer [chan pipeline-blocking]])
(let [output-chan (chan 100)
input-chan (chan 1000)]
(pipeline-blocking 4 ; parallelism knob
output-chan
(map unchangeable-lib-request)
input-chan)
;; Consume results from output-chan, put operations on input-chan
[output-chan input-chan]
)
This spawns n (in this case 4) threads that are kept busy executing unchangeable-lib-request.
Use the buffer size of output-chan to finetune how much requests you want to happen in advance.
Use the buffer size of input-chan to finetune how many requests you want scheduled without backpropagation (a blocking input-chan).
Related
I'm using SBCL 1.4.5 on Ubuntu 18.04.
It seems that the garbage collector in SBCL is not properly freeing up memory bound to hash tables with symbol keys and values. Interestingly, when the keys and values are integers, garbage collection works fine.
For example, the following program works correctly:
(defun copy-hash-table (hash)
(let ((h (make-hash-table
:test (hash-table-test hash)
:rehash-size (hash-table-rehash-size hash)
:rehash-threshold (hash-table-rehash-threshold hash)
:size (hash-table-size hash))))
(loop for key being the hash-keys of hash
using (hash-value value)
do
(setf (gethash key h) value)
finally (return h))))
(defun make-integer-input ()
(loop
with hash1 = (make-hash-table) and hash2 = (make-hash-table)
for i from 0 to 500
do
(setf (gethash (random 100) hash1) (random 100))
(setf (gethash (random 100) hash2) (random 100))
finally
(return (list hash1 hash2))))
(defun do-integer-work (hash1 hash2)
(loop
for i being the hash-keys of hash1
for j being the hash-keys of hash2
do
(remhash i hash1)
(setf (gethash i hash1) (random 100))
(remhash j hash2)
(setf (gethash j hash2) (random 100)))
(values hash1 hash2))
(defun hash-worker (list-obj)
(loop
with next
for i from 1 to 50000
do
(multiple-value-bind (new-hash1 new-hash2)
(do-integer-work (copy-hash-table (first list-obj)) (copy-hash-table (second list-obj)))
(setq next (list new-hash1 new-hash2))
(if (> (random 100) 50)
(setq list-obj next)))))
I ran this program by calling (hash-worker (make-integer-input)). The top-level function, hash-worker, takes in a list of two hash tables, and does work over a copy of the hash tables in do-integer-work. Then, then the helper function outputs two modified hash tables which are saved in new-hash1 and new-hash2. Then the system randomly decides to keep the modified hash tables or not.
The do-integer-work helper function sequentially removes the keys of the hash tables and resubmits them with new random values.
When this program runs, I observed that the memory consumption was basically constant for the duration of the program. This is not the case when I ran a sister program over hash tables with symbol keys and values.
(defun copy-hash-table (hash)
(let ((h (make-hash-table
:test (hash-table-test hash)
:rehash-size (hash-table-rehash-size hash)
:rehash-threshold (hash-table-rehash-threshold hash)
:size (hash-table-size hash))))
(loop for key being the hash-keys of hash
using (hash-value value)
do
(setf (gethash key h) value)
finally (return h))))
(defun make-symbol-input ()
(loop
with hash1 = (make-hash-table) and hash2 = (make-hash-table)
for i from 0 to 500
do
(setf (gethash (gentemp) hash1) (gentemp))
(setf (gethash (gentemp) hash2) (gentemp))
finally
(return (list hash1 hash2))))
(defun do-symbol-work (hash1 hash2)
(loop
for i being the hash-keys of hash1
for j being the hash-keys of hash2
do
(remhash i hash1)
(setf (gethash i hash1) (gentemp))
(remhash j hash2)
(setf (gethash j hash2) (gentemp)))
(values hash1 hash2))
(defun hash-worker (list-obj)
(loop
with next
for i from 1 to 50000
do
(multiple-value-bind (new-hash1 new-hash2)
(do-symbol-work (copy-hash-table (first list-obj)) (copy-hash-table (second list-obj)))
(setq next (list new-hash1 new-hash2))
(if (> (random 100) 50)
(setq list-obj next)))))
I ran this program calling (hash-worker (make-symbol-input)). The difference in this program is that the top-level function calls do-symbol-work. As this program ran, I observed the memory usage of the system steadily increase until my machine ran out of memory.
Is this a known bug in SBCL, and if so, is there a workaround for this?
You're using gentemp, which interns the symbol it creates. Such symbols can't be GCd because they are referenced by their package. So you are generating a huge number of interned symbols and killing the system. Instead, use gensym. There may still be GC bugs in code like this, but this isn't one.
I'm a beginner in Emacs lisp, so this is really a noob question. Let's say that I have to write a function that uses a loop to add 1 to each element of a numeric vector.
Here is the code I wrote (the comments indicate what I'm trying to do at each step):
(defun add-one (x)
"Use a loop to add 1 to each element of list X"
(let* ((res x) ; make a copy of X
(counter 0)) ; set counter to 0
(while (< counter (length x))
;; Replace each element of RES by adding 1:
(setcar (nthcdr counter res) (1+ (car (nthcdr counter x))))
(setq counter (1+ counter)))
;; Display result:
(message "%s" res)))
But my code seems to be destructive for x, since several calls to the function do not produce the same result:
;; Define a list:
(setq mylist '(1 2 3 4))
;; Several calls to the function:
(add-one mylist) ; -> (2 3 4 5)
(add-one mylist) ; -> (3 4 5 6)
Here is my question: I don't understand why my code is destructive (I expected the result to be (2 3 4 5) at each execution). I know that setcar is destructive, but it is applied to a copy of x, not to x itself. So why is the result changing?
Thanks!
For the sake of clarity:
Despite the fact you do not copy input list, your code is not at all lispy. Try this instead:
(mapcar '1+ '(1 2 3 4))
Let does not make a copy of anything, so this assigns the value referenced by the variable x to the variable res. Hence any changes to the list referenced by res also change the list referenced by x
(let* ((res x) ; make a copy of X
I'm making a function that generates a .wav file. I have the header all set, but I'm running into trouble with the data itself. I have a function for creating a sine wave at 880Hz (at least I think that's what it does, but that's not my question)--the question is, how do I convert a collection of byte arrays to just one byte array with their contents? This was my best try:
(defn lil-endian-bytes [i]
(let [i (int i)]
(byte-array
(map #(.byteValue %)
[(bit-and i 0x000000ff)
(bit-shift-right (bit-and i 0x0000ff00) 8)
(bit-shift-right (bit-and i 0x00ff0000) 16)
(bit-shift-right (bit-and i 0xff000000) 24)]))))
(def leb lil-endian-bytes)
(let [wr (io/output-stream (str name ".wav") :append true)]
(.write wr
(byte-array (flatten (concat (map
(fn [sample] (leb (* 0xffff (math/sin (+ (* 2 3.14 880) sample)))))
(range (* duration s-rate)) )))))
but it doesn't do what I want it to do: concat all of the byte-arrays into one vector and then into a single byte array. And it makes sense to me why it can't: it can't concat/flatten a byte[] because it's not a vector; it's a byte[]. And it can't cast a byte[] into a byte. But what do I need to do to get this working?
You might be looking for something like:
(byte-array (mapcat seq my-sequence-of-byte-arrays))
Converting the byte arrays to be concatenated to sequences and back to byte array may be a little inefficient if you are dealing with large arrays. Here's how to concatenate byte arrays making java.nio.ByteBuffer do the heavy lifting:
(defn concat-byte-arrays [& byte-arrays]
(when (not-empty byte-arrays)
(let [total-size (reduce + (map count byte-arrays))
result (byte-array total-size)
bb (java.nio.ByteBuffer/wrap result)]
(doseq [ba byte-arrays]
(.put bb ba))
result)))
It will be as simple as:
Your byte arrays that needs to be combined in a single byte-array:
(def byte-arrays [(byte-array 10 (byte 1))
(byte-array 10 (byte 2))
(byte-array 10 (byte 3))])
Combine:
(byte-array (for [ar byte-arrays
i ar] i))
I'll start with my test using the speclj framework.
(it "turns the string into a hash-map"
(should= {1 "1" 2 "2" 3 "3"}
(format-string "1=1 2=2 3=3")))
Then my code:
(:use [clojure.string :only (split)])
(defn format-string [string]
(split string #"\s+"))
Right now, the format-string function returns ["1=1" "2=2" "3=3"] and the test fails. As you can see in my test, I want it to return a hash-map with key-value pairs indicated by the = sign.
I've tried a few things and I've gotten close, but can't quite understand how to make this transformation.
EDIT
Figured out one solution, although the keys are strings instead of integers.
My code:
(defn format-board [route]
(let [[first second third] (split route #"\s+")]
(merge
(apply hash-map (split-at-equals first))
(apply hash-map (split-at-equals second))
(apply hash-map (split-at-equals third))
This returns {"1" "1" "2" "2" "3" "3"}.
You have split at the spaces, but then you need to split again at the = delimiter. You can use regular expressions to do the parsing. Once you have your pairs you can assoc into a hash-map. Here I've used reduce to effect the transformation.
user=> (reduce #(assoc % (read-string (nth %2 1)) (nth %2 2)) {}
#_> (re-seq #"([^=\s]+)=([^=\s]+)" "1=1 2=2 3=3") )
{3 "3", 2 "2", 1 "1"}
Note key order is not applicable to hash-maps
user=> (= {1 "1", 2 "2", 3 "3"} *1)
true
Here is the potential parallel version using clojure.core.reducers:
(require '[clojure.core.reducers :as r])
(require '[clojure.string :as s])
(def string-of-pairs "1=1 2=2 3=3 4=4")
; reducing fn to convert seq of (key, value) to hash-map
(defn rf ([] {}) ([acc [k v]] (assoc acc k v)))
; for large colls, fold will parallelize execution
(r/fold merge rf (r/map #(s/split % #"=") (s/split string-of-pairs #"\s+")))
For better understanding of reducers, watch this video where
Rich explains motivation behind reducers and demonstrates some usage.
I want to write some nice Math with DrRacket(R5RS but not only) (Racket Tag is a bit empty).
I would really like to code some matrix stuff like:
(3 3 3) (5 3 4)
(4 4 4) -> (5 3 4)
(5 5 5) (5 3 4)
And other stuff like this to set up some nice gimp filters...
Some folks pointed out, this could be done via lists inside lists, but I can't think of a practical example here...
I am looking forward to your reply.
Yours sincerely, Andreas_P
A few notes:
1) what do you mean with Scheme "is built in within the core"? GIMP now has a native Python-fu along with Script-fu
2) the Lisp that won Google AI Challenge is Common Lisp, not Scheme
3) I am not sure, but the Script-fu console states TinyScheme, thus I would expect it to be really essential, with little library support, with respect to more complete Scheme implementations
Anyway, I tried a few example on matrices "the Scheme way". For the sake of simplicity, they lack any control on the input data, but for simple examples they work fine on DrRacket.
(define (vect-sum x y)
(cond
((empty? x) empty)
(else
(cons (+ (first x) (first y)) (vect-sum (rest x) (rest y))))))
(define (scalar-prod a v)
(cond
((empty? v) empty)
(else
(cons (* a (first v)) (scalar-prod a (rest v))))))
(define (matr-sum x y)
(cond
((empty? x) empty)
(else
(cons (vect-sum (first x) (first y))
(matr-sum (rest x) (rest y))))))
(define (matr-scalar-prod a m)
(cond
((empty? m) empty)
(else
(cons (scalar-prod a (first m)) (matr-scalar-prod a (rest m))))))
And now a simple test on the data as in the other answer:
> (define m '((3 3 3)(5 3 4)(4 4 4)))
> m
'((3 3 3) (5 3 4) (4 4 4))
> (matr-scalar-prod 3 m)
'((9 9 9) (15 9 12) (12 12 12))
HHi, I'd highly recommend you to use Python instead of Scheme for writing GIMP scripts, unless you want to learn Scheme for recreational purposes.
One of the tenets in Python is to not allow the language to stay between you and your problem, and writing your own Matrix manipulation code is trivial. If you want high performance ops, you can use a third party library such as NumPy (even from inside GIMP environment) to get it.
So, for a Matrix class that would allow scallar multiplication and adding one could simply write:
class Matrix(object):
def __init__(self, rows, cols, *args):
self.rows = rows
self.cols = cols
self.values = args
def __getitem__(self, (i,j)):
return self.values[i * self.cols + j]
def __setitem__(self,(i,j), value):
self.values[i * self.cols + j] = value
def __add__(self, other):
values = []
for i in range(self.rows):
for j in range(self.cols):
values.append(self[i,j] + other[i,j])
return Matrix(self.rows, self.cols, *values)
def __mul__(self, N):
return Matrix(self.rows, self.cols,*(N * v for v in self.values))
def __repr__(self):
return "\n".join (" ".join(str(self[i,j]) for j in range(self.cols)) for i in range(self.rows) )
Example on Python's interactive console:
>>> m = Matrix(3,3,
... 3,3,3,
... 5,3,4,
... 4,4,4)
>>> m * 3
9 9 9
15 9 12
12 12 12
Implementing more operations is equally simple, and, for calling GIMP's API functions, with this example class, you can just use m.values, that is simply a list with all the matrix values in sequence - which is the way GIMP PDB's functions use them. (Such as pdb.gimp_drawable_transform_matrix or pdb.plug_in_convmatrix. (I suppose you've found GIMP's API browser under the help menu - the same functions are available in Scheme and in Python, just replace the "-" in the names for "_")