This is quite complex to explain, and a lot of code involve which I cannot paste here.
But I have a Test application which executes Test cases through designated plugins.
When the app is executed it creates a separate multi process called a writer, this module handles all updates to a web page which holds all running information about the test cases and their state.
For this write I also create an interface (WrIf). This interface hold a Queue to the Writer Thread together with a weakref.proxy().
Now when the Test app starts executing its Test cases it creates new multiprocessor from where it can call the specific plugin. That means that the WrIf is "serialized" to this multi-process.
For every call to the WrIf it makes a check to see if the main Writer thread is still running. however here is where I get a problem. I get the following assert when I try to call is_alive().
assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
I can expand some on this however I think it will get muddy fast because the app is rather large and somewhat complex.
Regards
Related
I have an application that has multiple screens and a process that needs to get UI info from some and update others.
Tried many methods but the result always is always "not a Java FX thread". Without using some kind of thread the UI does not update Because of the multi screen nature of the app (not practical to change) I need to fundamentally change the application architecture which is why I am not posting any code - its all going to change.
What I cant work out is the best way to do this and as any changes are likely to require substantial work I am reluctant to try something that has little chance of success.
I know about Platform.runLater and tried adding that to the updates but that was complex and did not seem to be effective.
I do have the code on GitHub - its a personal leaning project that started in Scala 2 but if you have an interest in learning or pointing out my errors I can provide access.
Hope you have enjoyed a wonderful Christmas.
PS just make the repo public https://github.com/udsl/Processor6502
The problem is not that the Platform.runLater was not working its because the process is being called form a loop in a thread and without a yield the JavaFX thread never gets an opportunity to run. It just appeared to be failing – again I fall foul of an assumption.
The thread calls a method from within a loop which terminates on a condition set by the method.
The process is planned to emulate the execution of 6502 processor instructions in 2 modes run and run-slow, run-slow is run with a short delay after each instruction execution.
The updates are to the main screen the PC, status flags and register contents. The run (debug) screen gets the current instruction display updated and other items will be added. In the future.
The BRK instruction with a zero-byte following is captures and set the execution mode to single-step essentially being a break point though in the future it will be possible via the debug screen to set a breakpoint and for the execution of the breakpoint to restore the original contents. This is to enable the debugging of a future hardware item – time and finances permitting – it’s a hobby after all 😊
It terns out that the JavaFX thread issue only happens when a FX control is written to but not when read from. Placing all reads and writes in a Platform.runLater was too complex which is why I was originally searching for an alternative solution but now only needed it protect the writes is much less a hassle.
In the process loop calling Thread.’yield’() enables the code in the Platform.runLater blocks to be executed on the JavaFX thread so the UI updates without an exception.
The code in the Run method:
val thread = new Thread {
override def run =
while runMode == RunMode.Running || runMode == RunMode.RunningSlow do
executeIns
Thread.`yield`()
if runMode == RunMode.RunningSlow then
Thread.sleep(50) // slow the loop down a bit
}
thread.start
Note that because yield is a Scala reserved word needs to quote it!
I'm writing a game in Rust where each player can submit some python scripts to the server in order to automate various tasks in the game. I plan on using pyo3 to run the python from rust.
However, I can see an issue arising if a player submits a script like this:
def on_event(e):
while True:
pass
Now when the server calls the function (using something like PyAny::call1()) the thread will hang as it reaches the infinite loop.
My first thought was to have pyo3 execute the python one statement at a time, therefore being able to exit if the script been running for over a certain threshold, but I don't think pyo3 supports this.
My next idea was to give each player their own thread to run their own scripts on, that way if one of their scripts got stuck it only affected their gameplay. However, I still have the issue of not being able to kill a thread when it gets stuck in an infinite loop - if a lot of players submitted scripts that just looped, lots of threads would start using a lot of CPU time.
All I need is way to execute python scripts in a way such that if one of them does loop, it does not affect the server's performance at all.
Thanks :)
One solution is to restrict the time that you give each user script to run.
You can do it via PyThreadState_SetAsyncExc, see here for some code. It uses C calls of the interpreter, which you probably can access in Rust (with PyO3 FFI magic).
Another way would be to do it on the OS level: if you spawn a process for the user script, and then kill it when it runs for too long. This might be more secure if you limit what a process can access (with some OS calls), but requires some boilerplate to communicate between the host.
I wonder how Camunda manage multiple instances of a sub-process.
For example this BPMN:
Let's say multi-instances process would iterate on a big collection, 500 instances.
I have a function in a web app that call the endpoint to complete the user common task, and perform another call to camunda engine to get all tasks (on first API call callback). I am supposed to get a list of 500 sub-process user tasks (the ones generated by the multi-instances process).
What if the get tasks call is performed before Camunda Engine successfully instantiated all sub-processes?
Do i get a partial list of task ?
How to detect that main and sub processes are ready?
I don't really know if Camunda is able to manage this problematic by itself so I though of the following solution, knowing I only can use Modeler environment with Groovy to add code (Javascript as well, but the entire code parts already added are groovy):
Use of a sub process throw event to catch in main process, then count and compare tasks ready with awaited tasks number for each signal emitted.
Thanks
I would maybe likely spawn the tasks as parallel process (or 500 of them) and then got to a next step in which I signal or otherwise set a state that indicates the spawning is completed. I would further join the parallel processes all together again and have here a task signaling or otherwise setting a state that indicates all the parallel processes are done. See https://docs.camunda.org/manual/7.12/reference/bpmn20/gateways/parallel-gateway/. This way you can know exactly at what point (after spawning is done and before the join) you have a chance of getting your 500 spawned sub processes
I have a test suite harness which is used to run test scripts (classes defined therein actually), and as it iterates through the tests, it manipulates the python logger such that the log messages are all output to different files, each associated with its own test (class). This works fine for tests run in a sequential manner where i can control the log handlers in the root logger which enable all log messages (from whatever libraries the test classes may use) to log their messages into the proper test log file.
But what I am really trying to figure out is how to run such tests in parallel (via threading or multiprocessing) such that each thread will have its own log file to place all such messages.
I believe that I still need to manipulate the root logger, because that is the only place both tests and the libraries they use will converge on to do all logging to a common place.
I was thinking that I could add a handler for each thread which would contain a log filter to only log from a particular thread, and that would get me close (haven't tried this yet, but seems possible in theory). And this would possibly be the full solution (if indeed such would work) except for one thing. I cannot tell test writers to not use threads themselves, in their tests. So if they did so, again, this solution would fail. I'm fine with test-internal threads all logging to the one file, but these new threads would fail to log to the file their parent thread is logging to. The filter doesn't know anything about them.
And I could be mistaken, but it seems that threading.Thread objects cannot determine their own parent thread? This precludes a better log handler filter that accepts messages generated in a thread or any of its child/descendant threads. (?)
Any suggestions about how to approach this would be great.
Thanks,
Bruce
I am building a simple application to download a set of XML files and parse them into a database using the async module (https://npmjs.org/package/node-async) for flow control. The overall flow is as follows:
Download list of datasets from API (single Request call)
Download metadata for each dataset to get link to XML file (async.each)
Download XML for each dataset (async.parallel)
Parse XML for each dataset into JSON objects (async.parallel)
Save each JSON object to a database (async.each)
In effect, for each dataset there is a parent process (2) which sets of a series of asynchronous child processes (3, 4, 5). The challenge that I am facing is that, because so many parent processes fire before all of the children of a particular process are complete, child processes seem to be getting queued up in the event loop, and it takes a long time for all of the child processes for a particular parent process to resolve and allow garbage collection to clean everything up. The result of this is that even though the program doesn't appear to have any memory leaks, memory usage is still too high, ultimately crashing the program.
One solution which worked was to make some of the child processes synchronous so that they can be grouped together in the event loop. However, I have also seen an alternative solution discussed here: https://groups.google.com/forum/#!topic/nodejs/Xp4htMTfvYY, which pushes parent processes into a queue and only allows a certain number to be running at once. My question then is does anyone know of a more robust module for handling this type of queueing, or any other viable alternative for handling this kind of flow control. I have been searching but so far no luck.
Thanks.
I decided to post this as an answer:
Don't launch all of the processes at once. Let the callback of one request launch the next one. The overall work is still asynchronous, but each request gets run in series. You can then pool up a certain number of the connections to be running simultaneously to maximize I/O throughput. Look at async.eachLimit and replace each of your async.each examples with it.
Your async.parallel calls may be causing issues as well.