I am working on a big project that puts performance as a high priority. I have a little bit of experience using wxPython to create windows and dialog boxes for software, but I have no experience in getting processes to work in parallel during the course of a single program.
So basically, what I want to accomplish is the following:
I want one main class that controls the high level program. It sets up a configuration either from a config file or from user input. This much I have accomplished on my own.
I need PROCESS #1 to read in a file and a list of commands, execute the commands, and then pass the modified file to PROCESS #2 (this requires that PROCESS #2 is ready to accept new input.) Once the file is passed, PROCESS #1 would begin work on the next set of inputs and wait for PROCESS #2 to finish before the cycle repeats.
PROCESS #2 takes input from PROCESS #1 and writes output to a log file. Once the output is complete, it waits for the next set of output from PROCESS #1.
I know how to use wxTimers and the events associated with that, but what I have found is that a timer event will not execute if the program is otherwise occupied (like in the middle of a method.)
I have seen threads about "threading" and "Pool", but the terminology tends to go over my head, and I haven't gotten any of that sort of stuff to work.
If anybody can point me in the right direction, I would be greatly appreciative.
If you use threads, then I think this would be fairly easy to do. Here's what I would suggest:
Create a button (or some other widget) to execute process #1 in a thread. The thread itself will run BOTH processes. Here's some psuedo-code that might help:
# this is in your thread code:
result = self.call_process_1(args)
self.call_process_2(result)
This will allow you to start another process #1/2 with a new set of commands every time you press the button. Since the two processes are encapsulated in the thread, they don't have to wait for process #2 to finish. You will probably need to log to separate logs for the logs to make sense, but you can label the logs with a timestamp and a thread number or a uuid.
Depending on how many of these processes you need to do, you might need to look into setting up a cluster that's driven with celery or some such. But I think this is a good starting place.
Related
I'm developing a game in pyglet, that scheduled by a simple text file like :
0:00:01;event1
0:00:02;event2
0:00:03;event3
The fact is that, among these events, some might be blocking (for instance event2 might consist in displaying instructions until a key is pressed). As a consequence, event3 might not be executed at the proper time (i.e., during the event2). For now, my strategy is to schedule one event after the other :
Execute the first event
Once the first event is finished, compute the remaining duration between the first and the second event (delta_duration)
Schedule the second event with a delay of delta_duration
... and so on
For now, I did not succeed in implementing properly a blocking event with this strategy. It seems that anything blocking the event_loop (like a sleep call during event2) is preventing even the graphical elements of event2 (text instructions) to be displayed. On the other hand, if I do not put any blocking routine (sleep) in the event2, I'm able to see the vertices, but the scheduler keeps on scheduling (!), and so the event3 comes too soon.
My question is : what would be a general strategy, in pyglet, to articulate non-blocking to blocking events ? More precisely, is it possible (desirable) to use multiple clocks for that purpose ? The pyglet documentation mentions that multiple clocks can be used but it is not very well explained.
I don't want a solution that is specific to my events example but, rather, general indications about the way to go.
It's really up to your program on what blocks. If you are using input from Python for the console window, then yes that will block because it's blocking execution of Python in general. If you have a label popup in the window that is waiting for input from an on_key_press window event, then that is completely different as it's not blocking the pyglet loop, it was scheduled within it.
If your event is a 20 second long math calculation, then that should probably be ran in a thread. You will probably have to separate the types of events in order to differentiate how they should be ran. It's hard to say because without a runnable example or sample of code, I am just guessing at your intentions.
Although it sounds more like you are wanting some sort of callback system. When execution of func1 is declared done, go to func2. There is nothing built into pyglet like this, you would have to have a clever use of scheduling. There are examples of this using pure python though. I personally use Twisted Deferred's for this.
I want to run a job on 10 threads to process 100 files. Each thread is supposed to work on separate file. When a thread is done it is supposed to pick the next file.
What I'm doing right now is basically going on a loop, kick off the job and make it run in the background (using &), wait for any process to end if the count of processes is greater than 10 and pick up the next file. It is working but is there a better to way to achieve this?
I don't see any better solution, as long as each file has to be processed separately.
You'd be better off having each thread process its file, then try to pick up the next file itself. Maybe it won't matter in your current application, but the starting and teardown of threads is relatively expensive. Common practice is to keep a thread alive if it is otherwise just going to be replaced by a clone of itself.
The program I am developing uses threads to deal with long running processes. I want to be able to use Gauge Pulse to show the user that whilst a long running thread is in progress, something is actually taking place. Otherwise visually nothing will happen for quite some time when processing large files & the user might think that the program is doing nothing.
I have placed a guage within the status bar of the program. My problem is this. I am having problems when trying to call gauge pulse, no matter where I place the code it either runs to fast then halts, or runs at the correct speed for a few seconds then halts.
I've tried placing the one line of code below into the thread itself. I have also tried create another thread from within the long running process thread to call the code below. I still get the same sort of problems.
I do not think that I could use wx.CallAfter as this would defeat the point. Pulse needs to be called whilst process is running, not after the fact. Also tried usin time.sleep(2) which is also not good as it slows the process down, which is something I want to avoid. Even when using time.sleep(2) I still had the same problems.
Any help would be massively appreciated!
progress_bar.Pulse()
You will need to find someway to send update requests to the main GUI from your thread during the long running process. For example, if you were downloading a very large file using a thread, you would download it in chunks and after each chunk is complete, you would send an update to the GUI.
If you are running something that doesn't really allow chunks, such as creating a large PDF with fop, then I suppose you could use a wx.Timer() that just tells the gauge to pulse every so often. Then when the thread finishes, it would send a message to stop the timer object from updating the gauge.
The former is best for showing progress while the latter works if you just want to show the user that your app is doing something. See also
http://wiki.wxpython.org/LongRunningTasks
http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/
http://www.blog.pythonlibrary.org/2013/09/04/wxpython-how-to-update-a-progress-bar-from-a-thread/
I'm implementing a simple job scheduler, which spans a new process for every job to run. When a job exits, I'd like it to report the number of actions executed to the scheduler.
The simplest way I could find, is to exit with the number of actions as a return code. The process would for example exit with return code 3 for "3 actions executed".
But the standard (AFAIK) being to use the return code 0 when a process exited successfully, and any other value when there was en error, would this approach risk to create any problem?
Note: the child process is not an executable script, but a fork of the parent, so not accessible from the outside world.
What you are looking for is inter process communication - and there are plenty ways to do it:
Sockets
Shared memory
Pipes
Exclusive file descriptors (to some extend, rather go for something else if you can)
...
Return convention changes are not something a regular programmer should dare to violate.
The only risk is confusing a calling script. What you describe makes sense, since what you want really is the count. As Joe said, use negative values for failures, and you should consider including a --help option that explains the return values ... so you can figure out what this code is doing when you try to use it next month.
I would use logs for it: log the number of actions executed to the scheduler. This way you can also log datetimes and other extra info.
I would not change the return convention...
If the scheduler spans a child and you are writing that you could also open a pipe per child, or a named pipes or maybe unix domain sockets, and use that for inter process communication and writing the processed jobs there.
I would stick with conventions, namely returning 0 for success, expecially if your program is visible/usable around by other people, or anyway document well those decisions.
Anyway apart from conventions there are also standards.
I'm running a long process in the background. I've managed to output the console data to gui. But the problem is that, the data is returned only after the process is finished. But I need to display the data at realtime. ie, I need to display the data, every time it produces some output on the console. I'm running the process with in my gui from a seperate thread.
I mean, it would be like building a gui for the ping command, where output is displayed on console after each packet is send, ie at realtime. I just need to redirect that to gui, in realtime. I'm implementing the gui in wxwidgets. Any help would be greatly appreciated.
Thanking You..
Jvc
Is the output you wish to display generated in a separate process from the process running the GUI? Or in a separate thread in the same process?
I ask because most people, when they ask this question, mean a a separate thread. Since you have tagged your question with "process" I will assume that is what you mean.
You need some inter-process communication. There is a bewildering variety of techniques to do this. Personally, I always use sockets.
wxWidgets has simple, easy to use socket classes wxSocketClient and wxSocketServer.
The background process is probably not running wxWidgets, so you will need something else there. I reccomend boost::asio. I know it looks intimidating, but in fact the tutorial code can be used as is.
There is a lot more to be said, but I risk straying away from the point, since there are so few details in your question.
You can have an output queue protected by a wxMutex. The thread doing the computation writes to the queue, then signals the GUI thread using wxQueueEvent with a custom event to let it know that the thread is not empty. The GUI thread then reads the queue and outputs the data.