What is the best way of solving the simultaneity bias in groovy?
I have a issue with a queue. I select all queue with state wait and
update the state to running. Most of the time it goes okay, but sometimes by error state running get loaded from another thread.
I know from study time, I get illustrated to a way to solve through java. But I can't remember how.
Below should illustrate the problem.
Thread one:
List<Queue> elements = Queue.findAll { state = wait }
sleep(1000) //Illustration of a delay
elements.each {
it.state = running
it.save()
}
Thread two (starts at same time as thread one):
List<Queue> elements = Queue.findAll { state = wait } //The error occur here, as thread one already begin executing same elements.
//Problem should be solved somehow here.
elements.each {
it.state = running
it.save()
}
Thread two select same element as Thread one. But thread two should not go further with the element, as Thread one is the owner now.
I have a function which receives tasks and puts them into a channel. Every task has ID, some properties and a channel where result will be placed. It looks like this
task.Result = make(chan *TaskResult)
queue <- task
result := <-task.Result
sendReponse(result)
Another goroutine takes a task from the channel, processes it and puts the result into task's channel
task := <-queue
task.Result <- doExpensiveComputation(task)
This code works fine. But now I want to coalesce tasks in the queue. Task processing is a very expensive operation, so I want process all the tasks in the queue with the same IDs once. I see two ways of doing it.
First one is not to put tasks with the same IDs to the queue, so when existing task arrives it will wait for it's copy to complete. Here is pseudo-code
if newTask in queue {
existing := queue.getById(newTask.ID)
existing.waitForComplete()
sendResponse(existing.ProcessingResult)
} else {
queue.enqueue(newTask)
}
So, I can implement it using go channel and map for random access + some synchronization means like mutex. What I don't like about this way is that I have to carry both map and channel around the code and keep their contents synchronized.
The second way is to put all the tasks into queue, but to extract task and all the tasks with the same IDs from the queue when result arrives, then send result to all the tasks. Here is pseudo-code
someTask := queue.dequeue()
result := doExpensiveComputation(someTask)
someTask.Result <- result
moreTasks := queue.getAllWithID(someTask.ID)
for _,theSameTask := range moreTasks {
theSameTask.Result <- result
}
And I have an idea how to implement this using chan + map + mutex in the same way as above.
And here is the question: is there some builtin/existing data structures which I can use for such a problem? Are there another (better) ways of doing this?
If I understand the problem correctly, the simplest solution that comes into my mind is adding a middle layer between task senders (putting into queue) and workers (taking from queue). This, probably routine, would be responsible for storing current tasks (by ID) and broadcasting the results to every matching tasks.
Pseugo code:
go func() {
active := make(map[TaskID][]Task)
for {
select {
case task := <-queue:
tasks := active[task.ID]
// No tasks with such ID, start heavy work
if len(tasks) == 0 {
worker <- task
}
// Save task for the result
active[task.ID] = append(active[task.ID], task)
case r := <-response:
// Broadcast to all tasks
for _, task := range active[r.ID] {
task.Result <- r.Result
}
}
}
}()
No mutexes needed and probably no need to carry anything around either, workers will simply need to put all the results into this middle layer, which is then routing responses correctly. You could even easily add caching here if there's a chance clashing IDs can arrive some time apart.
Edit: I had this dream where the above code caused a deadlock. If you send a lot of requests at once and choke worker channel there's a serious problem – this middle layer routine is stuck on worker <- task waiting for a worker to finish, but all the workers will be probably blocked on send to response channel (because our routine can't collect it). Playable proof.
One could think of adding some buffers into the channels but this is not a proper solution (unless you can design the system in such way the buffer will never fill up). There're a few ways of solving this problem; for example, you can run a separate routine for collecting responses, but then you would need to protect active map with a mutex. Doable. You could also put worker <- task into a select, which would try to send task to a worker, receive new task (if nothing to send) or collect a response. One could take advantage of the fact that nil channel is never ready for communication (ignored by select), so you can alternate between receiving and sending tasks within a single select. Example:
go func() {
var next Task // received task which needs to be passed to a worker
in := queue // incoming channel (new tasks) -- active
var out chan Task // outgoing channel (to workers) -- inactive
for {
select {
case t := <-in:
next = t // store task, so we can pass to worker
in, out = nil, worker // deactivate incoming channel, activate outgoing
case out <- next:
in, out = queue, nil // deactivate outgoing channel, activate incoming
case r := <-response:
collect <- r
}
}
}()
play
I have a strange issue with multi-threading. I want to print a table view and therefore start a new thread which runs with a progress bar. Eventually this thread dies with memory errors for which I'm seeking the cause. Right now I got
malloc: * error for object 0x10000078c: Invalid signature for
pointer dequeued from free list
set a breakpoint in malloc_error_break to debug CaLister(27054,0x7fff73ea3300) malloc: error for object
0x60800043bcc0: Invalid pointer dequeued from free list
* set a breakpoint in malloc_error_break to debug
But most of the times (still it happens rarely!) it just stops with no specific (but definitely memory related) error. When I look from the debugger my data are nil. But they have not been touched since they are still being available for display in my table view.
Now the question: is there any precaution I need to take to access data being allocated in the main thread so I can safely access them from the detached thread?
Edit:
My print thread is dispatched like this (stripped code):
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0)) { self.performPrint (true) }
func performPrint (async:Bool) {
//First get the shared print info object so we know page sizes. The shared print info object acts like a global variable.
let sharedPrintInfo = NSPrintInfo.sharedPrintInfo()
let printObject = PSPrint ()
//Allocate a new instance of NSView into the variable printPageView
var frame = NSRect(x: 0, y: 0, width: sharedPrintInfo.paperSize.width-sharedPrintInfo.leftMargin-sharedPrintInfo.rightMargin, height: sharedPrintInfo.paperSize.height-sharedPrintInfo.topMargin-sharedPrintInfo.bottomMargin)
let basePrintPageView = PSPrintView(frame: frame)
var printPageView:PSPrintView
for pageNo in 1...paperDimensions.pages {
paperDimensions.pageNo = pageNo
printPageView = basePrintPageView.clone(pageNo)
//Set the option for the printView for what it should draw.
paperDimensions.pageNo = pageNo
//Finally append the view to the PSPrint Object.
printObject.printViews.append(printPageView)
}
dispatch_async(dispatch_get_main_queue()) {
printObject.printTheViews() //print all the views, each view being a 'page'.
}
}
Within
printPageView = basePrintPageView.clone(pageNo)
the access to my table view (where I get the data to be printed) returns nil sometimes.
Edit2: I just noticed that it's not the background thread which crashed, but the main thread :-/ Scratching my head even more but likely I have to close this question.
I have created two threads in DigitalMicrograph and they are executed as soon as the script is executed.
I want something different.
Let's imagine two buttons for the threads (start and stop thread).
How can I add code to activate the threads only when I push buttons ?
It would be very helpful if you had a code exemple for me.
There are a couple of things to consider:
You can not allocate new objects from within the UIframe object. (To be more precise: From a method invoked by a UI action. You can allocate f.e. in the constructor or with an Init() method at start.) Therefore you allocate them beforehand and then let the UIframe object know about them.
You often want the UIframe object to be aware of the thread object, but also the thread-object to be aware of the UIframe object. (So that the UI can change if something in the thread object wants it to.)
Having objects as member variables of objects is a bit dangerous, because those objects can only be released once the 'keeping' object gets released to. If two objects hold each-other as member variables, you're in a dead-locked situation! For this reason, it is save to use weak referencing: Keep only the objectID numbers as member variables and look objects up on need.
The following code-example should give you a starting point. It consists of 2 classes and a main call. The code is split in this answer, just copy & paste it into a single script file for testing.
First the thread object:
class myThread:Thread
{
number linkedDLG_ID
number externalBreak
myThread( object self )
{
result( self.ScriptObjectGetID() + " created.\n" )
}
~myThread( object self )
{
result( self.ScriptObjectGetID() + " destroyed.\n" )
}
void SetLinkedDialogID( object self, number ID ) { linkedDLG_ID = ID; }
void InterruptAtNextChance( object self ) { externalBreak = 1; }
void RunThread( object self )
{
number maxLoop = 30
object callDLG = GetScriptObjectFromID( linkedDLG_ID )
externalBreak = 0
for( number i=0; i<maxLoop; i++ )
{
sleep( 0.1 )
Result( i + "\n" )
if ( callDLG.ScriptObjectIsValid() )
{
callDLG.DLGSetProgress( "progress", (i+1)/maxLoop )
callDLG.ValidateView()
}
if ( externalBreak )
break;
}
// Cleanup at end of thread
if ( callDLG.ScriptObjectIsValid() )
{
callDLG.DLGSetProgress( "progress", 0 )
callDLG.LookUpElement( "DBevel" ).DLGValue( 0 )
callDLG.ValidateView( )
}
}
}
Any threading class is derived from the class Thread.
The class has two member variables. One will hold the ID of the UI-object, the other is a simple Boolean to allow 'outside' calls to stop a running thread.
The first two methods are the constructor and the destructor. They are not really needed in this example, but it is good practice to put them in during script development, because they will indicate in the results-window when an object of that class gets created and when it gets destroyed. That helps tracking memory leaks and dead-lock situations.
The next two methods allow 'outside' calls to set the two member variables.
The RunThread method is the heart of any Thread class. It has to be of exactly this signature because it overrides the according method of the parent class Thread from which we derive our class MyThread. The RunThread method gets launched into a separate background thread, when the method StartThread() is called. ( StartThread() is a method of the class Thread. )
The actual code in RunThread is in two parts:
An 'action-loop' doing anything you want but allowing a quick-exit if the Boolean variable changes value. This is how external calls can interrupt. This is discussed a bit further down.
A 'clean-up' part where the object can influence the UI object, discussed below as well.
Next is the UI class:
class myDialog:UIframe
{
object callThread
myDialog( object self )
{
result( self.ScriptObjectGetID() + " created.\n" )
}
~myDialog( object self )
{
result( self.ScriptObjectGetID() + " destroyed.\n")
}
TagGroup CreateDLG( object self )
{
image i := IntegerImage( "", 1, 0, 25, 25)
i = 0; i[ 2 , 2 , 23 , 23 ] = 1;
image onImage, offImage
onImage = RGB( 0*i , 200*i , 0*i )
offImage = RGB( 200*i , 0*i , 0*i )
TagGroup tgItems, tg, button, label, progress
tg = DLGCreateDialog("Dialog",tgItems)
button = DLGCreateDualStateBevelButton( "DBevel", onImage, offImage, "StartPressed" )
progress = DLGCreateProgressBar( "progress" ).DLGfill( "X" )
label = DLGCreateLabel( "start/stop" )
tgItems.DLGAddElement( DLGGroupItems( button , label ).DLGTableLayout( 2 , 1 , 0 ) )
tgItems.DLGAddElement( progress )
return tg
}
object Init(object self, number callThreadID )
{
// Assign thread-object via weak-reference
callThread = GetScriptObjectFromID( callThreadID )
if ( !callThread.ScriptObjectIsvalid() )
Throw( "Invalid thread object passed in! Object of given ID not found." )
// Pass weak-reference to thread object
callThread.SetLinkedDialogID( self.ScriptObjectGetID() )
return self.super.init( self.CreateDLG() )
}
void StartPressed( object self )
{
number active = self.LookupElement( "DBevel" ).DLGGetValue()
if ( active )
callThread.StartThread()
else
callThread.InterruptAtNextChance()
}
}
Any dialog (UI) class is derived from the class UIframe.
This class has only one member variable: An object, which will be the thread-object.
Again there are a constructor/destructor method for easier debugging.
The CreateDLG method builds the tagGroup describing the dialog. I will not go into details here, but essentially it creates the following dialog when displayed:
The Init() method initializes the object. The Init() method of the base class UIframe requires a descriptive TagGroup and returns the UI object itself. We call on this in the last line of our extended Init() method, and use our class-method to create the tagGroup:
return self.super.init( self.CreateDLG() )
The code before is what links our thread-object to the UI-object. We pass in a number, which is the object-ID of our thread-object. We now get the according object from memory and assign it to our local member variable. (NB: The variable now holds the object itself, not a copy or clone of it! )
callThread = GetScriptObjectFromID( callThreadID )
Right away, we check if the lookup was successful and actually returned a valid object. If not, we stop our script with a thrown exception. From now on, the UI-object 'contains' the thread-object and can use it.
Now comes the back-link. Now that the UI object has been allocated, it also has an object-ID. We feed this number into our thread-object.
callThread.SetLinkedDialogID( self.ScriptObjectGetID() )
From now on, the thread object is nicely linked to the UI-object. Looking back to the myThread class, you will notice that we use the same trick of looking up and locally storing the object in the RunThread() method.
StartPressed() is the method linked to our dialog button. This button is a bevel button, so we query its state, which is the state after the bevel-button changed, and act accordingly. We either launch the RunThread() method of our thread object as a background-thread, or invoke the according 'interrupt' method, which simply sets the Boolean variable
Finally the main script:
void StartScript()
{
object threadObject = alloc( myThread )
object dlgObject = alloc( myDialog ).Init( threadObject.ScriptObjectGetID() )
dlgObject.display( "Dialog" )
}
StartScript()
Not a lot going on here. We first create the threadObject of the myThread class, and then the dialog UI object.
We initialize the dialog object with the ID of the existing threadObject, and then display it as a modeless dialog.
Some points to notice:
Whenever you use object variables in DigitalMicrograph scripting, you should put them into a structure block. This ensures that the objects get out-of-scope and deleted, when the structure block is left. Object variables defined and allocated in the main script are not destructed at the end of the script. For this reason, we have encapsulated the main script in a method itself.
We have used two different methods of linking in this example:
Direct: The myDialog class really keeps the thread-object itself as a member variable. Although we initialized it with the ID only, we immediately linked the object to a member variable.
Weak reference: The myThread class only holds the object-ID of the dialog-object as a member variable.
Why have we done this? If the myThread class would keep the dialog-object as a member, then the two objects would hold each-other in a dead-lock situation. Neither can be destructed because of the other. But why have we not used the same for the myDialog class? Because we want to display the dialog as a modeless dialog in a background thread itself!
Think of the main-script:
We create the thread-object
We create the dialog-object
We display the dialog-object (But we don't stop script execution here!)
The script ends
But when the script ends, the object variables threadObject and dlgObject go out of scope! They will be immediately destructed, unless something keeps them in memory. The dlgObject stays in memory, because we displayed it as modeless dialog. It will be released, when the according dialog window is closed. But what keeps the threadObject? Nothing! Once the RunThread() method has finished, it would be released and subsequently destructed. However, because it is a member of the dlgObject it will not be destructed.
In my qt application I'm printing some large html code from a QWebFrame (mainFrame from a QWebPage). The html code contains some inline images and takes a while to be printed. Therefore I tried to put the print call into a separate thread to keep the gui responsive. As mentioned in the docs I thought this should be possible (Painting in threads).
But I still get the famous "QPixmap: It is not safe to use pixmaps outside the GUI thread"
My code to print is as follows:
void PrintDialog::paintRequested(QPrinter *printer) {
futureWatcher = new QFutureWatcher<void>();
QEventLoop q;
connect(futureWatcher, SIGNAL(finished()), &q, SLOT(quit()), Qt::UniqueConnection);
futureWatcher->setFuture(QtConcurrent::run(m_webPage->mainFrame(), &QWebFrame::print, printer));
q.exec();
}
To clarify all objects reside in the gui thread and the html gets generated before. The images get inlined like follows:
QImage image
QByteArray ba;
QBuffer buffer(&ba);
buffer.open(QIODevice::WriteOnly);
image.save(&buffer, "PNG");
QString html = "<img src=\"data:image/png;base64," + QString(ba.toBase64() + "\"/>";
So what am I doing wrong here? Is it that the QWebFrame internally uses some QPixmaps? Is there any solution for using the QWebFrame::print call in another thread?