I've been trying to track down a problem with uwsgi where the uwsgi process kills itself.
The oh-so-helpful log files just say...
F*CK !!! i must kill myself (pid: 9984 app_id: 0)...
A little Googling led me to this line in the source code...
void harakiri() {
uwsgi_log("\nF*CK !!! i must kill myself (pid: %d app_id: %d)...\n", uwsgi.mypid, uwsgi.wsgi_req->app_id);
//Some other stuff
exit(0);
}
Whether it dies or not varies but seems (from Googling) to be tied to how long a request takes. In this instance, the request is streaming back a dynamically generated Pdf. The generation happens in the background but once it's complete, a new request comes in to retrieve it. The Pdf can be potentially quite large (worst-case, 50-60MB) which - depending on the connection - speed explains why requests might reach a timeout threshold.
How can I configure uwsgi to either never time out or have extremely high timeouts? The app is being used on private networks and I'd rather it was slow and succeeded than died.
harakiri is something you voluntary enable with --harakiri, by default there is no such feature. Check your configuration for it.
Another possibility could be you are running without the master process (you should have a warning about it) and set an alarm() without defining a signal handler for SIGALRM
Related
I am dealing with an odd problem which I couldn't find the answer to online, nor through a lot of trial and error.
In a multi-multi process cluster, forked worker processes can run arbitrarily long commands, but the parent process listens for keepalive messages sent by workers, and kills workers that are stuck for longer than X seconds.
Worker processes can asynchronously communicate with the rest of the world (using http, or process.send ipc communication), but on exit, I'd like to be able to communicate some things (typically, queued logs or error details).
Most online documentation for process.on('exit', handler) indicates usage of console.log, however it seems like forked processes don't inherit a normal stdout, and the console.log isn't a direct tty, it's a stream (the ipc stream, I presume?).
Because of this, the process exit handler doesn't let me use console.log to log extra lines (or if it does, I'm not sure where these lines end up)
I tried various combinations of fork options (silent/not silent, non-default stdio options like inherit), using fs.write to write to tty or a real file, using process.send, or but in no case, was I able to get the on-exit handler to log anywhere visible.
How can I get the forked process to successfully log on exit?
small additional points - all this testing is on unix-like systems (macos , amazon linux...) and both parent and child processes are fired with --sigint-trace so that we can get at least the top 10 stack frames of the interrupted process on exit. These frames do make it out to the terminal successfully
This was a bit of a misunderstanding about how SIGINT is handled, and I believe that it's impossible to accomplish what I want here, but I'd love to hear if someone else found a solution.
Node has its own SIGINT handler which is "more powerful" than custom SIGINT handlers - typically it interrupts infinite loops, which is extremely useful in the case where code is blocked by long-running operations.
Node allows one-upping its own SIGINT debugging capabilities by attaching a --trace-sigint flag which captures the last frames of execution.
If I understood this correctly, there are 4 cases with different behavior
No custom handler, event loop blocked
process is terminated without any further code execution. (and --trace-sigint can give a few stack traces)
No custom handler, event loop not blocked
normal exit flow, process.on('exit') event fires.
Custom handler, event loop blocked
nothing happens until event loop unblocks (if it does), then normal exit flow
Custom handler, event loop not blocked
normal exit flow.
This happens regardless of the way the process is started, and it's not a problem about pipes or exit events - in the case where the event loop is blocked and the native signal handler is in place, the process terminates without any further execution.
It would seem like there is no way to both get a forced process exit during a blocked event loop, AND still get node code to run on the same process after the native interruption to recover more information.
Given this, I believe the best way to recover information from the stuck process is to stream data out of it before it freezes (sounds obvious, but brings a lot of extra considerations in production environments).
I have a C++ application on Mac OS X. The app runs an event processing with the glfw library on the main thread and reads input and execute commands on a background C++ std::thread.
I am observing a frustrating phenomenon that I cannot explain so far.
If I make a call to a long running function on the background thread, initially that thread is using 100% of a core. But, after it has used a few seconds of CPU (10 seems to be the magic threshold), it gets throttled down to 25%.
If I start a computation run on a thread in the background before starting the glfw event processing loop (the event processing is essentially stuck waiting for events, as I don't even open a window), then it can use 100% for as long as it wants.
My biggest problem is that I have no idea what could be causing this nor how to figure it out. I've tested retrieving the pthread sched_param and changing the sched_priority from what seems to be default 31 to various values between 20 and 60 and it does not help.
I have identified one more condition for the phenomenon to happen:
The background thread has to have read from the terminal. It happens when I run the following background thread and enter a line for the computation to take place:
std::thread cmd([argc, argv, &scriptingRunner] {
for (std::string line; std::getline(std::cin, line); ) {
longComputation();
}
Perhaps App Nap is throttling your application to save energy. To check, open the Activity Monitor program and right-click on the header of the processes table to bring up the context menu, and click on "App Nap" in the context menu to enable the App Nap column; then look at your process in the table and see if its value in the App Nap column switches to "Yes" when the fault occurs.
If you want to disable app nap for your app, see the code listed in the question here.
my problem is the following: In my Flask-Restx-Application I created a Runner-Thread which runs asynchronously to the main-thread of the Flask-Application.
Now when I do changes as usual the Debugger still shows * Detected change in 'XXXXX', reloading which is a useful feature. The problem is that now it got stuck and cannot reload because of the running Thread which must be stopped manually.
I would still like to use the automatic reload if possible in combination with the asynchronous Runner-Thread. Is there a possibility to "detect" those reloads by triggering an Event or something similar? Then I could manually shutdown the Runner-Thread and restart it with the application. Or is there at least a possibility to not block the reload in order to proceed reloading the flask-restx-related stuff?
Thanks in advance for any help.
PS: I find it hard to add code here because I do not know which parts of the flask-app are important. If you need any code to answer the question I will add it in an Edit.
You need to make your thread a daemon thread if you want the reloader to work. The reloader will try to kill and restart the program (by killing the main thread), but because your other thread is not a daemon, it will fail to kill the program and reload it. A daemon thread is one that only lives as long as the main thread lives, and therefore making your other thread a daemon will fix your issue. Here is an example:
from threading import Thread
...
t = Thread(...)
t.daemon = True # this makes your thread a daemon thread
t.start()
I need to fetch the values of about 200 sensors every 15 seconds or so. To fetch the values I simply need to make an HTTP call with basic authentication and parse the response. The catch is that these sensors might be on slow connection so I need to wait at least 5 seconds for one sensor (but usually they respond a lot quicker, but there's always some that are slow and timeout).
So right now I have the following setup for that:
There is a NodeJS process that is connected to my DB and knows all about the sensors. It checks regularly to see if there are new ones or there are some that got deleted. It spawns a child process for every sensor, and in case the child process dies it restarts it. Also it kills it if the sensor gets deleted. The child process makes the HTTP call to its sensor with a 5 second timeout value and if it receives the value, saves it to Redis. Also it is in an infinite loop with a 15 seconds setTimeout. And there is a third process that copies all the values from Redis to the main MySQL DB.
So that has been a working solution for half a year, but after a major system upgrade (from Ubuntu 14.04 to 18.04 and thus every package upgraded as well) it seems to leak some memory and I can't seem to figure out where.
After starting out, the processes summarised take about 1.5GB of memory. But after a day or so this goes up to 3GB and so on and before running out of memory I need to kill all node processes and restart the whole thing.
So now I am trying to figure out more efficient methods to achieve the same result (query around 2-300 URLs every 15 sec and store the result in MySQL). At the moment I'm thinking of ditching Redis and the child processes will communicate with their master process and the master process will write to MySQL directly. This way I don't need to load the Redis library into every child process and that might save me some time.
So I need ideas on how to reduce memory usage for that application (I'm limited to PHP and NodeJS, mainly because of my knowledge, so writing a native daemon might be out of the question)
Thanks!
The solution was easier than I thought. I had to rewrite the child process into a native bash script and that brought down the memory usage to almost being zero.
I want to run a small clean up process every few hours on an Erlang server.
I know of the timer module. I saw an example in a tutorial used chained timer:sleep commands to wait for an event that would occur multiple days later, which I found strange. I understand that Erlang process are unique compared to those in other languages, but the idea of a process/thread sleeping for days, weeks, and even months at a time seemed odd.
So I set out to find out the details of what sleeping actually does. The closest I found was a blog post mentioning that sleep is implemented with a receive timeout, but that still left the question:
What do these sleep/sleep-like functions actually do?
Is my process taking up resources as it sleeps? Would having thousands of sleeping process use as many resources, as say, thousands of process servicing a recursive call that did nothing? Is there any performance penalty from repeatedly sleeping within processes, or sleeping for long periods of time? Is the VM constantly expending resources to see if the conditions to end the processes' sleep are up?
And as a side note, I'd appreciate if someone could comment on if there is a better way than sleeping to pause for hours or days at a time?
That is the Karma of any erlang process: it waits or dies :o)
when a process is spawned, it start executing until the last execution line, and die, returning the last evaluation.
To keep a process alive, there is no other solution to recursively loop in a never ending succession of calls.
of course there are several conditions that make it stop or sleep:
end of the loop: the process received a message which tell him to
stop recursion
a receive bloc: the process will wait until a message
matching one entry in the receive bloc is posted in the message
queue.
The VM scheduler stop it temporarily to let access to the CPU
to other processes
in the 2 last cases the execution will restart under the responsibility of the VM scheduler.
while waiting it uses no CPU bandwidth, but keeps the exact same memory layout it had when it started waiting. The Erlang OTP offers some means to reduce this memory layout to the minimum using the hibernate option (see the documentation of gen_serevr or gen_fsm, but it is for advanced usage only in my mind).
a simple way to create a "signal" that will fire a process at regular (or almost regular) interval is effectively to use receive block with timout (The timeout is limited to 65535 ms), for example:
on_tick_sec(Module,Function,Arglist,Period) ->
on_tick(Module,Function,Arglist,1000,Period,0).
on_tick_mn(Module,Function,Arglist,Period) ->
on_tick(Module,Function,Arglist,60000,Period,0).
on_tick_hr(Module,Function,Arglist,Period) ->
on_tick(Module,Function,Arglist,60000,Period*60,0).
on_tick(Module,Function,Arglist,TimeBase,Period,Period) ->
apply(Module,Function,Arglist),
on_tick(Module,Function,Arglist,TimeBase,Period,0);
on_tick(Module,Function,Arglist,TimeBase,Period,CountTimeBase) ->
receive
stop -> stopped
after TimeBase ->
on_tick(Module,Function,Arglist,TimeBase,Period,CountTimeBase+1)
end.
and usage:
1> Pid = spawn(util,on_tick_sec,[io,format,["hello~n"],5]).
<0.40.0>
hello
hello
hello
hello
2> Pid ! stop.
stop
3>
[edit]
The timer module is a standard gen_server running in a separate process. All the function in the timer module are public interfaces that execute a hidden gen_server:call or gen_server:cast to the timer server. This is a common usage to hide the internal of a server and allow further evolutions without impact on existing applications.
The server uses internally a table (ets) to store all the actions it has to do along with each timer reference and it uses its own function to be awaken when needed (at the end, the VM must take care of this ?).
So you can hibernate a process without any effect on the timer server behavior. The hibernation mechanism is
tricky, see documentation at hibernate/3 definition, you will see that yo have to "rebuild" the context by yourself since everything was removed from the process context, and a tuple(Module,Function,Arguments} is stored by the system to restart your process when needed.
cost some time in garbage collecting and process restart
It is why I said that it is really an advance feature that need good reason to be used.
There is also erlang:hibernate/3 that puts a process in "deep sleep", minimizing memory usage for it.