I have a custom logging class for my Python script with a flush() method which print()s the contents of a list.
I would like to include flush() in the special __del__() method in case the program ends without the log being flushed. However a note in the documentation states:
[...] when del() is invoked in response to a module being deleted (e.g., when execution of the program is done), other globals referenced by the del() method may already have been deleted or in the process of being torn down (e.g. the import machinery shutting down).
Would anyone recommend a different way of doing this, and if so, why?
You might want to look into making this logger a context manager. That still will not flush in the case of abnormal termination, but few things will. But __del__ might not be called on objects even in normal termination.
Loggers might be one of the things that doesn't fit well when using the with statement, as they are quite global, so it's not sure context manager is a good fit.
Related
So I'm writing a script/application that uses pythons multiprocessing BaseManager class. Now for the most part it works great, the only issue I have is that I am using the serve_forever as a blocking statement and then continue onwards however when I want to terminate or exit out of the serve_forever() function(ality) it automatically exits out and terminates the application, but like I mentioned I have some more things I want to take care of before I completely exit out.
I can exit out of serve_forever() by setting a stop event with stop_event.set(). Now this is all well and dandy however according to the source (https://github.com/python/cpython/blob/3.6/Lib/multiprocessing/managers.py#L147) serve_forever explicitly states sys.exit(0) and is part of the Server class that BaseManager uses within it's definition. Essentially I would like to remove that line (sys.exit(0)). How would I accompolish this?
When I search I'm coming up with results such as monkey patching? Can I just Subclass the Server class, explicitly define serve_forever to be the exact same code but without the sys.exit(0) line and call it a day? Something tells me that is not going to work. Do I subclass Server AND BaseManager?
Thanks!
Attempting to monkey-patch or inherit internal classes will result in code that will not be compatible across Python releases, not even patches.
Atop of that, these solutions will be unnecessarily complex and complicated, and are overall frowned upon.
I highly suggest re-implementing serve_forever() by using the start() method together with an event. Waiting for the event to be called or, if impossible, a loop checking if the manager is still alive, will be much easier and a better solution in almost all aspects that I can think of.
After discussing in chat, we realised the easiest approach is to just suppress the SystemExit being thrown from sys.exit(). I'm opening a bug report on CPython bug tracker accordingly to prevent sys.exit(). Do keep in mind the server will not actually shut down as it is run on a different thread. The whole recommendation of using .server().serve_forever() in the stdlib looks dubious at best.
If you wish to immediately shut down the server, call Server.listener.close() after catcing the exception.
This is what says on http://invisible-island.net/ncurses/ncurses.faq.html#multithread
If you have a program which uses curses in more than one thread, you will almost certainly see odd behavior. That is because curses relies upon static variables for both input and output. Using one thread for input and other(s) for output cannot solve the problem, nor can extra screen updates help. This FAQ is not a tutorial on threaded programming.
Specifically, it mentions it is not safe even if input and output are done on separate threads. Would it be safe if we further use a mutex for the whole ncurses library so that at most one thread can be calling any ncurses function at a time? If not, what would be other cheap workarounds to use ncurses safely in multi-thread application?
I'm asking this question because I notice a real application often has its own event loop but relies on ncurses getch function to get keyboard input. But if the main thread is block waiting in its own event loop, then it has no chance to call getch. A seemingly applicable solution is to call getch in a different thread, which hasn't caused me a problem yet, but as what says above is actually not safe, and was verified by another user here. So I'm wondering what is the best way to merge getch into an application's own event loop.
I'm considering making getch non-blocking and waking up the main thread regularly (every 10-100 ms) to check if there is something to read. But this adds an additional delay between key events and makes the application less responsive. Also, I'm not sure if that would cause any problems with some ncurses internal delay such as ESCDELAY.
Another solution I'm considering is to poll stdin directly. But I guess ncurses should also be doing something like that and reading the same stream from two different places looks bad.
The text also mentions the "ncursest" or "ncursestw" libraries, but they seem to be less available, for example, if you are using a different language binding of curses. It would be great if there is a viable solution with the standard ncurses library.
Without the thread-support, you're out of luck for using curses functions in more than one thread. That's because most of the curses calls use static or global data. The getch function for instance calls refresh which can update the whole screen—using the global pointers curscr and stdscr. The difference in the thread-support configuration is that global values are converted to functions and mutex's added.
If you want to read stdin from a different thread and run curses in one thread, you probably can make that work by checking the file descriptor (i.e., 0) for pending activity and alerting the thread which runs curses to tell it to read data.
I'm using process.on('uncaughtException') to catch any exceptions that would unexpectedly come up. In the function I write the data to a file, send an email, in the future it will likely do more.
Is there a way I can encapsulate the process.on() event in a file, then somehow require it in all the files that make up the application so I don't need to add that chunk of code to each file?
Node normally runs in a single process, so you only need process.on('uncaughtException') in one place.
The exception is if you use the cluster module or otherwise spawn other node processes, in which case you need the process.on('uncaughtException') loaded once for each process, but still not once for each file.
(Be careful doing too much in this handler, because by this point the process is considered unstable. I'm also not sure if async work is guaranteed to be run. The docs say that the correct use of 'uncaughtException' is to perform synchronous cleanup of allocated resources.)
I have python filesystem (watchdog) code successfully triggering a pyqtSignal#emit() call from its own thread (proven by a pdb trace).
This signal should be wired via connect(...) to a corresponding #pyqtSlot function on my own QObject, but the target function is in fact never called when a filesystem change triggers the emit.
The example code can be inspected at
https://github.com/cefn/xmlorgmode/blob/2b97ff9994132def035d325fc7f7095c9fe187f2/index.py
and can be invoked as follows, with an XML file and an XQuery file passed in from the same folder.
python index.py index.xml index.xq
This loads perfectly the first time, but no refresh ever takes place as triggered by the filesystem, which is what I would expect if my code worked. Failure is silent.
Can anyone suggest the additional event loop structures which I need for the watchdog-triggered emit to correspond to a call to QueryDisplay#update(...) ?
BACKGROUND
I've been experimenting with passing dynamically constructed HTML to QWebView backed by text files in a folder which change occasionally. Because of the requirement that QWebView#setHtml(...) is called within its own thread, I've had to try and figure out signals and slots, and I've hit a brick wall. The threading arrangement that I've arrived at is somehow broken but I can't figure out how.
I cannot establish how my QObject is supposed to be associated with an event loop within the appropriate thread so that things emitted can get handled. Having debugged the running code, the QApplication, QWebView and my QObject all share the same thread when calling #thread() interactively with pdb to establish affinity, just before entering the app.exec_() loop. I thought this would mean that all eventing for all these objects would be handled within app.exec_() but I must have misunderstood the model somehow.
(Pdb) adaptor.thread()
<PyQt4.QtCore.QThread object at 0xb30c3c44>
(Pdb) display.view.thread()
<PyQt4.QtCore.QThread object at 0xb30c3c44>
(Pdb) app.thread()
<PyQt4.QtCore.QThread object at 0xb30c3c44>
This was PEBKAC, but with at least two interesting feature interactions. Essentially the emit() was indeed triggering the slot but all the attempts I'd made to monitor the triggering were flawed.
First of all, PDB simply ignores breakpoints if they don't occur in the main thread. They are never triggered, even if those lines of code are actually executed, if it happens in another thread. This is extremely bad news, but worse, is a silent failure. A workaround is to manually change the code to invoke PDB inline, rather than use the debugger shell, but that kind of defeats the purpose of having a debugger to investigate the flow of control. For this reason I didn't see the invocation actually happening even with a breakpoint in place.
Secondly, QXMLQuery silently ignores changes to files and caches automatically, so even if it's reinvoked, and the underlying 'focus' files have actually changed, it ignores the changes. For this reason I couldn't see the results of the interaction being triggered (a cache prevented the change appearing).
I want to write a few asserts around a complicated multithreaded piece of code.
Is there some way to do a
assert(GetCurrentThreadId() == ThreadOfCriticalSection(sec));
If you want to do this properly I think you have use a wrapper object around your critical sections which will track which thread (if any) owns each CS in debug builds.
i.e. Rather than call EnterCriticalSection directly, you'd call a method on your wrapper which did the EnterCriticalSection and then, when it succeeded, stored GetCurrentThreadId in a DWORD which the asserts would check. Another method would zero that thread ID DWORD before calling LeaveCriticalSection.
(In release builds, the wrapper would probably omit the extra stuff and just call Enter/LeaveCriticalSection.)
As Casablanca points out, the owner thread ID is within the current CRITICAL_SECTION structure, so using a wrapper like I suggest would be storing redundant information. But, as Casablanca also points out, the CRITICAL_SECTION structure is not part of any API contract and could change. (In fact, it has changed in past Windows versions.)
Knowing the internal structure is useful for debugging but should not be used in production code.
So which method you use depends on how "proper" you want your solution to be. If you just want some temporary asserts for tracking down problems today, on the current version of Windows, then using the CRITICAL_SECTION fields directly seems reasonable to me. Just don't expect those asserts to be valid forever. If you want something that will last longer, use a wrapper.
(Another advantage of using a wrapper is that you'll get RAII. i.e. The wrapper's constructor and destructor will take care of the InitializeCriticalSection and DeleteCriticalSection calls so you no longer have to worry about them. Speaking of which, I find it extremely useful to have a helper object which enters a CS on construction and then automatically leaves it on destruction. No more critical sections accidentally left locked because a function had an early return hidden in the middle of it...)
As far as I know, there is no documented way to get this information. If you look at the headers, the CRITICAL_SECTION structure contains a thread handle, but I wouldn't rely on such information because internal structures could change without notice. A better way would be to maintain this information yourself whenever a thread enters/exits the critical section.
Your requirement doesn't make sense. If your current thread is not the thread which is in the critical section, then the code within the current thread won't be running, it'll be blocked when trying to lock the critical section.
If your thread is actually inside the critical section, then your assertion will always be true. If it's not, your assertion will always be false!
So what I mean is, assuming you're able to track which thread is in the critical section, if you place your assertion inside the critical section code, it'll always be true. If you place it outside, it'll always be false.