Background:
I am using C#, but I think this question applies to Java and Scala as well. I have one master actor that passes along all "Job Messages". These jobs can be executed at the same time, as long as they don't share don't share a "concurrency id". To take care of this, I was spawning one child actor for each "concurrency id". I then used setReceiveTimeout to clean up the children when they've been idle.
What I can't figure out is how I can do this without a race condition. Here are a couple of ways that don't work:
1) Remove the child from the Dictionary, and tell the child to terminate. This does not work because messages might have been added to the child's queue between the time the timeout was fired, and before the parent started processing the message, resulting in messages being lost.
2) Remove the child from the Dictionary, and send a PoisonPill. This does not work because the child might continue to process new work. If more work reaches the parent, the parent would create a new child, and the means two thing with the same concurrency id are running at the same time.
3) Remove the child from the dictionary, and "ask" the child with a message. If the message returns
Is there any way the parent can ask the child if there are any messages in the child's queue? (This would not be a race condition, because all messages to the child come from the parent)
Can the parent check if the child's queue is empty, and the child isn't processing anything?
Should I add a new message and "ask" the child if it's done? (This would be safe because I know the parent is the only one who sends messages, but there's a moderate chance this might block if the child is processing a message, or if the there's no threads available in the dispatcher's pool.
My question is similar to this question, but I'm adding the additional constraint of the "concurrency id", and not worried about "zombie actors" who are currently shutting down, as long as the zombies have no work and won't get more work:
Get or create child Akka actor and ensure liveness
Akka makes sure that actor mailboxes are private. The mailbox of one actor is no business of another actor. A parent checking if the child's mailbox is empty would be wrong.
Why do you think ask would block?
One option: do not kill the children. If you have a fixed set of concurrency IDs you use over and over again and again, just keep all the actors alive. Actors do not consume too much resources.
Another option:
When the parent wants to kill the child:
send a PoisonPill to the child and remove him from the Dictionary.
When the parent receives a "job message" with concurrencyID:
if (Dictionary.contains(concurrencyID) {
send message to child
} else {
if (parent has child with name concurrencyID) {
delay message - for example with scheduler // child is terminating
} else {
create a child with name concurrencyID
send message to child
}
}
Related
import os
from kazoo.client import KazooClient
signal.signal(signal.SIGINT, signal_handler)
def signal_handler(signal,frame):
print('\nYou pressed Ctrl+C!')
print("Stopping...Process "+process_id)
print()
children = zk.get_children("/assign2/root")
l = [int(i[3:]) for i in children]
l.remove(int(process_id))
print("Min process id : ",min(l))
zk.delete("/assign2/root/pid"+process_id, recursive=True)
#To run a command on terminal --> os.system("python3 zook.py")
if(int(process_id)==min(l)):
print("Starting new process through :" , min(l))
os.system("python3 zook.py")
os.kill(int(process_id),0)
zk.stop()
sys.exit(0)
zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()
zk.ensure_path("/assign2/root")
zk.create("/assign2/root/pid"+process_id, bytes(process_id, encoding='utf-8'),ephemeral=True)
On killing a process, I want to find the smallest of the remaining pids and start a process through the smallest pid.
I am using zookeeper to store pids as ephemeral znodes and upon terminating the python script in terminal, I'm trying to create another process.
I am able to create a process through the pid of the process I am terminating, but not from another pid of my choice.
Please let me know if there is a way to do this.
So, what you need here is monitoring all your different processes (each process monitoring all other), and wait for the failure of a process. In zookeeper world, this can be achieved via:
1) Zookeeper watches: Each zookeeper client (process) can set up a watch on the parent znode (In your case /assign2/root/) and watch for CHILD_DELETE events. You can have a look at the zookeeper documentation and the documentation of your specify library (kazoo) to see how you can do that. On setting up the watch, you can specify a callback which would be executed asynchronously, each time a znode under your parent znode disappears. This could happen because:
The child znode is ephemeral and the zk client which create that
znode died.
The client explicitly deleted the child node. For example in your case you can delete the child node in the signal handler.
In this callback, each of the alive zookeeper clients/processes, will determine if its the lowest ranked process id (by fetching the list of all existing children under the parent znode), and if it is indeed the smallest pid, it will spawn the new process (python3 zook.py).
Note, that zookeeper watches are one-time fire concept. This means that after the watch has fired (i.e. the callback has executed), you would want to reset the watch at the very end of the callback execution, so that the next child delete event will also result in the callback firing.
2) The alternative approach is polling. In each of your process, you can have a dedicated thread which can periodically monitor the children of your parent znode, and each time a process detects that a node has disappeared, it can again determine if its the lowest alive pid and spawn a new process, if it indeed is.
Hope this helps. Let me know if you have any doubts. Would have liked to post the code you needed, but have to run for an errand.
I am trying to persist data into database. My persist method is asynchronous.
class MyActor(persistenceFactory:PersistenceFactory) extends Actor {
def receive: Receive = {
case record: Record =>
// this method is asynchronous, immediate return Future[Int]
persistenceFactory.persist(record)
}
}
The bottleneck is here either we get out of memory or no thread available when the application runs under increased load.
So what is the best way to handle asynchronous calls inside receive method of Akka actor ?
This is a great example of when you should have an actor create another actor to handle an interaction. Basically the flow is like this and you can use FSM (finite state machine) too if it makes it easier for you.
Parent actor receives message
Parent actor creates child transaction actor
Parent sends the child the original message
Child calls async method and stores the Future
Child uses become() to change its behavior to wait for completion of the future. There are a number of ways to do this, including FSM.
Child schedules periodic messages to itself to check the Future
The Child's new behavior checks the Future if it is complete on each receive
When the Future is complete, child can reply with the result to the parent or to the original sender (if the parent fwded the message with the original sender).
I was looking through the pipe(2) syscall example in linux, I got this from tldp: http://tldp.org/LDP/lpg/node11.html#SECTION00722000000000000000
When we need to close the input of child process we close fd(1) of child - fine, but we should also close the output of the parent i.e. close fd(0) of parent, why should we use else statement here, in this case the parent's fd(0) will close only when the fork fails, am I correct?
I feel there should not be else statement and both the input of child and output of parent should be closed for communication from child to parent correct?
You shouldn't talk about child input and parent output, that looks like you are referring to stdin and stdout, which is not necessarily the same as the pipe's read and write channels.
For communication from child to parent, the child needs to close the pipe's read channel (fd[0] in your example), and the parent needs to close the pipe's write channel (fd[1]).
Your confusion seems to be more about forking than about pipes.
The else is needed, because we need to execute different code in the parent and in the child. It is very common to use if / else after forking to differentiate the code that executes in each process. Remember that fork(2) returns twice: in the parent, and in the newborn child. It returns the child's pid in the parent, and 0 in the child, so we use that to tell them apart.
In the example you posted, if fork(2) fails, the first if is entered and the process exits. Otherwise, a pair of if / else is used to execute different code in each process.
I have a form that is responsible for creating and setting up an instance of an object, and then telling the object to go do its work. The process is a long one, so there's an area on the form where status messages appears to let the user know something is happening. Messages are set with a setMessage(string msg) function. To allow the form to remain responsive to events, I create a new thread for the object to run in, and pass it the setMessage function as a delegate to allow the object to set status messages on the form. This part is working properly. The main form is responsive and messages posted to its setMessage function appear as expected.
Because the process is a long one, and is made up of many steps, I want to allow the user to terminate the process before it's finished. To do this I created a volatile bool called _stopRequested and a function called shouldStop() that returns its value. This is also given to the object as a delegate. The object can tell if it should terminate by checking shouldStop() periodically, and if it's true, shut down gracefully.
Lastly, Windows controls are not thread safe, so the compiler will complain if a thread other than the one that created the control tries to manipulate it. Therefore, the setMessage function is wrapped in an if statement that tests for this and invokes the function using the parent thread if it's being called from the worker thread (see http://msdn.microsoft.com/en-us/library/ms171728(v=vs.80).aspx for a description).
The problem arises when the user requests a shutdown. The main form sets _stopRequested to true and then waits for the child thread to finish before closing the application. It does this by executing _child.Join(). Now the parent thread (the one running the form) is in a Join state and can't do anything. The child thread (running the long process) detects the stop flag and attempts to shut down, but before it does, it posts a status message by calling it's setMessage delegate. That delegate points back to the main form, which figures out that the thread setting the message (child) is different than the thread that created the control (parent) and invokes the function in the parent thread. The parent thread is, of course, in a Join state and won't set the text on the text box until the child thread terminates. The child thread won't terminate because it's waiting for the delegate it called to return. Instant deadlock.
I've found examples of signaling a thread to terminate, and I've found examples of child threads sending messages to the parent thread, but I can't find any examples of both things happening at the same time. Can someone give me some pointers on how to avoid this deadlock? Specifically, I'd like the form to wait until the child thread terminates before closing the application but remain able to do work while it waits.
Thanks in advance for the advice.
1-(lazy) Dispatch the method from a new Thread so it doesn't lock
2-(re-think) The main UI thread should be able to control the child thread, so forget the _stopRequested and shouldStop() and implement a childThread.Abort() , abort does not kill the thread, but sends a ThreadAbortException
which can be handled or even canceled
catch(ThreadAbortException e)
{
ReleaseResources();
}
Make the ReleaseResources safe by making various checks such as:
resource != null
or
resource.IsClosed()
The ReleaseResources should be called normally without abort and also by abort.
3-(if possible)stop the child, via main thread call ReleaseResources()
You may have to implement a mix of these.
int main(){
fork();
}
I know this is a newbie question, but my understanding is that the parent process now will fork a new child process exactly as the parent one, which means that the child should also fork a child process and so on... In reality, this only generates one child process. I cant understand what code will the child be executing?
The child process begins executing at the exact point where the last one left off - after the fork statement. If you wanted to fork forever, you'd have to put it in a while loop.
As everybody mentioned, the child also starts executing after fork() has finished. Thus, it doesn't call fork again.
You could see it clearly in the very common usage like this:
int main()
{
if (fork())
{
// you are in parent. The return value of fork was the pid of the child
// here you can do stuff and perhaps eventually `wait` on the child
}
else
{
// you are in the child. The return value of fork was 0
// you may often see here an `exec*` command
}
}
You missed a semi-colon.
But the child (and also the parent) is continuing just after the fork happenned. From the point of view of application programming, fork (like all system calls) is "atomic".
The only difference between the two processes (which after the fork have conceptually separate memory spaces) is the result of the fork.
If the child went on to call fork, the child would have two forks (the one that created it and the one that it then made) while the parent would only have one (the one that gave it a child). The nature of fork is that one process calls it and two processes return from it.