I want to ensure a code block within a JSR223 Post Processor is executed by only one thread at a time. The JSR223 Post Processor is placed just below the Test Plan component to ensure it is shared with all the samplers.
I have tried with following to check if synchronization works as expected
synchronized (this){
def count=props.get("count").toInteger()
count+=1
props.put("count",count.toString())
log.info(" ==== Current count is ${count} ${__threadNum} ${vars.get('__jm__TG__idx')}=== ")
//different code to be used within this block
}
I have noted duplicates (i.e. duplicate count values)
props is a global object so when multiple threads are writing into it the previous value is getting overwritten
When you're inlining JMeter Functions or Variables like ${__threadNum} into Groovy scripts only first value is getting cached and used in the subsequent iterations
Each thread is executing JSR223 test elements separately, sychronized keyword doesn't act as you expect
So the options are in:
Either convert your JSR223 PreProcessor into JSR223 Sampler and put it under Critical Section Controller (however it will mean "no concurrency"
Or go for Inter-Thread Communication Plugin
You may include JSR Sampler in Critical Section Controller to ensure it's synchronized on JVM level
ensures that its children elements (samplers/controllers, etc.) will be executed by only one thread as a named lock will be taken before executing children of controller.
Critical Section Controller takes locks only within one JVM, so if using Distributed testing ensure your use case does not rely on all threads of all JVMs blocking.
I have a main thread where I am getting a list of categories from there I am passing each category from inside the ForEach Controller to next thread. In this thread I get a number of categories inside the variable
In the second thread I am going through each category and using ForEach Controller to pass each product into the 3rd Thread. In this thread I get a number of products inside the variable.
Now I want to use the above number of categories and products into the second and third threads as a dynamic variable i.e if:
categories are 10 then the second thread numbers should be 10
products are 100 then I would like 100/10 = 10 threads for 3 thread group.
I am using the interthread communication processors successfully which works fine withe static number of threads but not when I pass as a variable via interthread communication processors
Please help me
In first Thread Group define the desired number of threads using __setProperty() function like:
${__setProperty(threads,10, )}
In second Thread Group read the value using __P() function like:
${__P(threads,)}
That's it, 2nd Thread Group will kick off as many threads as you define in the first one
More information: Knit One Pearl Two: How to Use Variables in Different Thread Groups
Also be aware that since JMeter 3.1 you should be using JSR223 Test Elements and Groovy language for any form of scripting so convert your Beanshell test elements into JSR223 and make sure to use Groovy
Please, i got a problem with Love2d thread functions, and i didnt find any examples or explanations i would understand anywhere.
first:
in main file i got:
thread1 = love.thread.newThread("ambient.lua")
thread2 = love.thread.newThread("ambient.lua")
thread1:start()
thread2:start()
ambient.lua contains:
random1 = love.math.random(1, 10)
gen1 = love.audio.newSource("audio/ambient/gen/a/gen_a_".."random1"..".mp3",
"static")
gen1:setVolume(1.0)
gen1:setLooping(false)
gen1:play()
works fine, problem is that when i ask var = Thread1:isRunning( ) in same step or with delay, when audio is playing and try to print it, it throws error (is supposedly null). when the audio finishes, i see that memory is cleared. also if i link thread1:start() to mouse click and then start it multiple times in short time, memory usage goes up like crazy, then after time similar to sample length it starts to decrease. question is, am i creating multiple threads? in that case did they even terminate properly after samples ended? or is the thread lifetime just 1-step long and i am only creating multiple audio sources playing with the same thread? is problem in the check itself?
next issue is that i need to use thread1:start() with values:
thread1:start(volume, sampleID)
and i have no clue how to addres them in the thread itself. guides and examples says "vararg" reference. i didnt see any decent explanation or any example containing "..." usage in variable input into threads. i am in need of example how to write it. even if this audio fiddle is not a great example, i will surely need it for AI. No need for complicated input, just simple x,y,size,target_x,target_y values.
and i have no clue how to addres them in the thread itself. guides and
examples says "vararg" reference. i didnt see any decent explanation
or any example containing "..." usage in variable input into threads
You didn't read manuals enough. Every loaded Lua chunk (section 2.4.1 of Lua 5.1 manuals) is an anonymous function with variable number of arguments.When you call love.thread.newThread("ambient.lua"), Love2D will create new chunk, so basic Lua rules applies to this case.
In your example, volume and sampleID parameters from within thread would be retrieved like this:
local volume, sampleID = ...
gen1 = love.audio.newSource(get_stream_by_id(sampleID), "static")
gen1:setVolume(volume)
gen1:setLooping(false)
gen1:play()
I am implementing some functionality which requires me to implement an API to wait for d3d10 to finish rendering.Basically i am trying to implement synchronize access to a shared texture such that i can update a texture after d3d10 is successfully able to present to backbuffer. By calling this black box api i think this can be achieved and i think it will be something similar like glfinish().I have read we can use ID3D10Query queries for implementing synchronize access.
D3D10_QUERY_DESC queryDesc;
... // Fill out queryDesc structure
ID3D10Query * pQuery;
pDevice->CreateQuery( &queryDesc, &pQuery );
pQuery->Begin();
... // Issue graphis commands, do whatever
pQuery->End();
UINT64 queryData; // This data type is different depending on the query type
while( S_OK != pQuery->GetData( &queryData, sizeof( UINT64 ), 0 ) )
{
}
should i put some dummy command between begin and end this? since i want to expose this functionality as public API something named as waitforgraphiscompletion
what should be a dummy command here?
If you are trying to synchronize execution CPU and GPU in OpenGL, you would use glFenceSync to followed by a glClientWaitSync. The equivalents in Direct 10 are ID3D10Asynchronous::End, and ID3D10Asynchronous::GetData (note, in DX11, the interfaces are slightly different). These let you know when the GPU has finished processing the command buffer to a particular point. This allows you to know when previous read/write operations on a resource have completed, and the CPU can safely access the resource without additional synchronization.
You are not required to put any commands in the while loop. The command buffer will eventually process your query, and will return S_OK (or an error message, which you might want to handle). However, this is somewhat wasteful, as the CPU will just spin waiting for the GPU, so if possible, you should do some extra useful work within the loop.
Note, if you used D3D10_ASYNC_GETDATA_DONOTFLUSH as the final parameter to GetData (instead of '0'), the above would not be the case - there is no guarantee that the command buffer would 'automatically' kick off, and you could end up in an infinite loop (and, as such is not the recommended usage).
Coding in Lua, I have a triply nested loop that goes through 6000 iterations. All 6000 iterations are independent and can easily be parallelized. What threads package for Lua compiles out of the box and gets decent parallel speedups on four or more cores?
Here's what I know so far:
luaproc comes from the core Lua team, but the software bundle on luaforge is old, and the mailing list has reports of it segfaulting. Also, it's not obvious to me how to use the scalar message-passing model to get results ultimately into a parent thread.
Lua Lanes makes interesting claims but seems to be a heavyweight, complex solution. Many messages on the mailing list report trouble getting Lua Lanes to build or work for them. I myself have had trouble getting the underlying "Lua rocks" distribution mechanism to work for me.
LuaThread requires explicit locking and requires that communication between threads be mediated by global variables that are protected by locks. I could imagine worse, but I'd be happier with a higher level of abstraction.
Concurrent Lua provides an attractive message-passing model similar to Erlang, but it says that processes do not share memory. It is not clear whether spawn actually works with any Lua function or whether there are restrictions.
Russ Cox proposed an occasional threading model that works only for C threads. Not useful for me.
I will upvote all answers that report on actual experience with these or any other multithreading package, or any answer that provides new information.
For reference, here is the loop I would like to parallelize:
for tid, tests in pairs(tests) do
local results = { }
matrix[tid] = results
for i, test in pairs(tests) do
if test.valid then
results[i] = { }
local results = results[i]
for sid, bin in pairs(binaries) do
local outcome, witness = run_test(test, bin)
results[sid] = { outcome = outcome, witness = witness }
end
end
end
end
The run_test function is passed in as an argument, so a package can be useful to me only if it can run arbitrary functions in parallel. My goal is enough parallelism to get 100% CPU utilization on 6 to 8 cores.
Norman wrote concerning luaproc:
"it's not obvious to me how to use the scalar message-passing model to get results ultimately into a parent thread"
I had the same problem with a use case I was dealing with. I liked lua proc due to its simple and light implementation, but my use case had C code that was calling lua, which was triggering a co-routine that needed to send/receive messages to interact with other luaproc threads.
To achieve my desired functionality I had to add features to luaproc to allow sending and receiving messages from the parent thread or any other thread not running from the luaproc scheduler. Additionally, my changes allow using luaproc send/receive from coroutines created from luaproc.newproc() created lua states.
I added an additional luaproc.addproc() function to the api which is to be called from any lua state running from a context not controlled by the luaproc scheduler in order to set itself up with luaproc for sending/receiving messages.
I am considering posting the source as a new github project or contacting the developers and seeing if they would like to pull my additions. Suggestions as to how I should make it available to others are welcome.
Check the threads library in torch family. It implements a thread pool model: a few true threads (pthread in linux and windows thread in win32) are created first. Each thread has a lua_State object and a blocking job queue that admits jobs added from the main thread.
Lua objects are copied over from main thread to the job thread. However C objects such as Torch tensors or tds data structures can be passed to job threads via pointers -- this is how limited shared memory is achieved.
This is a perfect example of MapReduce
You can use LuaRings to accomplish your parallelization needs.
Concurrent Lua might seem like the way to go, but as I note in my updates below, it doesn't run things in parallel. The approach I tried was to spawn several processes that execute pickled closures received through the message queue.
Update
Concurrent Lua seems to handle first-class functions and closures without a hitch. See the following example program.
require 'concurrent'
local NUM_WORKERS = 4 -- number of worker threads to use
local NUM_WORKITEMS = 100 -- number of work items for processing
-- calls the received function in the local thread context
function worker(pid)
while true do
-- request new work
concurrent.send(pid, { pid = concurrent.self() })
local msg = concurrent.receive()
-- exit when instructed
if msg.exit then return end
-- otherwise, run the provided function
msg.work()
end
end
-- creates workers, produces all the work and performs shutdown
function tasker()
local pid = concurrent.self()
-- create the worker threads
for i = 1, NUM_WORKERS do concurrent.spawn(worker, pid) end
-- provide work to threads as requests are received
for i = 1, NUM_WORKITEMS do
local msg = concurrent.receive()
-- send the work as a closure
concurrent.send(msg.pid, { work = function() print(i) end, pid = pid })
end
-- shutdown the threads as they complete
for i = 1, NUM_WORKERS do
local msg = concurrent.receive()
concurrent.send(msg.pid, { exit = true })
end
end
-- create the task process
local pid = concurrent.spawn(tasker)
-- run the event loop until all threads terminate
concurrent.loop()
Update 2
Scratch all of that stuff above. Something didn't look right when I was testing this. It turns out that Concurrent Lua isn't concurrent at all. The "processes" are implemented with coroutines and all run cooperatively in the same thread context. That's what we get for not reading carefully!
So, at least I eliminated one of the options I guess. :(
I realize that this is not a works-out-of-the-box solution, but, maybe go old-school and play with forks? (Assuming you're on a POSIX system.)
What I would have done:
Right before your loop, put all tests in a queue, accessible between processes. (A file, a Redis LIST or anything else you like most.)
Also before the loop, spawn several forks with lua-posix (same as the number of cores or even more depending on the nature of tests). In parent fork wait until all children will quit.
In each fork in a loop, get a test from the queue, execute it, put results somewhere. (To a file, to a Redis LIST, anywhere else you like.) If there are no more tests in queue, quit.
In the parent fetch and process all test results as you do now.
This assumes that test parameters and results are serializable. But even if they are not, I think that it should be rather easy to cheat around that.
I've now built a parallel application using luaproc. Here are some misconceptions that kept me from adopting it sooner, and how to work around them.
Once the parallel threads are launched, as far as I can tell there is no way for them to communicate back to the parent. This property was the big block for me. Eventually I realized the way forward: when it's done forking threads, the parent stops and waits. The job that would have been done by the parent should instead be done by a child thread, which should be dedicated to that job. Not a great model, but it works.
Communication between parent and children is very limited. The parent can communicate only scalar values: strings, Booleans, and numbers. If the parent wants to communicate more complex values, like tables and functions, it must code them as strings. Such coding can take place inline in the program, or (especially) functions can be parked into the filesystem and loaded into the child using require.
The children inherit nothing of the parent's environment. In particular, they don't inherit package.path or package.cpath. I had to work around this by the way I wrote the code for the children.
The most convenient way to communicate from parent to child is to define the child as a function, and to have the child capture parental information in its free variables, known in Lua parlances as "upvalues." These free variables may not be global variables, and they must be scalars. Still, it's a decent model. Here's an example:
local function spawner(N, workers)
return function()
local luaproc = require 'luaproc'
for i = 1, N do
luaproc.send('source', i)
end
for i = 1, workers do
luaproc.send('source', nil)
end
end
end
This code is used as, e.g.,
assert(luaproc.newproc(spawner(randoms, workers)))
This call is how values randoms and workers are communicated from parent to child.
The assertion is essential here, as if you forget the rules and accidentally capture a table or a local function, luaproc.newproc will fail.
Once I understood these properties, luaproc did indeed work "out of the box", when downloaded from askyrme on github.
ETA: There is an annoying limitation: in some circumstances, calling fread() in one thread can prevent other threads from being scheduled. In particular, if I run the sequence
local file = io.popen(command, 'r')
local result = file:read '*a'
file:close()
return result
the read operation blocks all other threads. I don't know why this is---I assume it is some nonsense going on within glibc. The workaround I used was to call directly to read(2), which required a little glue code, but this works properly with io.popen and file:close().
There's one other limitation worth noting:
Unlike Tony Hoare's original conception of communicating sequential processing, and unlike most mature, serious implementations of synchronous message passing, luaproc does not allow a receiver to block on multiple channels simultaneously. This limitation is serious, and it rules out many of the design patterns that synchronous message-passing is good at, but it's still find for many simple models of parallelism, especially the "parbegin" sort that I needed to solve for my original problem.