I am creating a queue system where I need a set of workers to run jobs. For the queue I am planning on using Amazon's SQS. For my workers I would like to run instances of Codeigniter because the core application will also use Codeigniter and I would like to share the models of the workers with that of the core application.
I see that it is possible to hit a controller method using CI's CLI (http://codeigniter.com/user_guide/general/cli.html). However, these workers should continually run, unless I kill them. I have looked into this article about setting up daemons using php: http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/ This method requires that I use this class: https://github.com/kvz/system_daemon.
I am not sure if I should be calling and running System Daemon inside my controller in CI to spawn the entire CI instance as a daemon? Or is there a command-line approach where I can establish my call to the CI method as a daemon and then avoid using the 'System Daemon' class altogether? Any considerations I should be aware of? I have never created a running daemon, I have only used the cron tab to restart processes but that will not work in my case.
I dont know if anyone has daemonized properly codeigniter (I read that many had tried).
I had a similar situation with yours and I ended up using the System Daemon class which constantly checks SQS and then calling a CLI codeigniter via system() passing the message as a parameter.
My approach works for ~10 months without having any issues and serves ~30-40k messages per day.
Related
I've got an application w. AppHost (inside Startup.cs) and also a Configure.Db.cs file.
I want to run a background service (Timer based) to routinely do some things, in addition to serving services.
If I add the timer inside Apphost's Configure(), which is run before the Configure.Db, I get into a problem as the background service is dependent on the Db to be configured.
Edit: I just found that it is possible to specify order of Modular startup: https://docs.servicestack.net/modular-startup#modular-startup-prioritization which means I can force the Configure.db to run before Apphost.
Anyway, am I going about this wrong? Should I perhaps have a separate AppHost just for the background job?
I would recommend configuring a Background MQ Service for executing Services in the background.
You can look at how to do this with hangfire by adding the mix
x mix hangfire-postgres
This will allow you to add services that fire in the background on a cron schedule.
To make it work without a database you need to change the postgres provider to the in-memory provider: https://github.com/perrich/Hangfire.MemoryStorage
We have multiple Google Cloud Run services running for an API. There is one parent service and multiple child services. When the parent service starts it loads a schema from all the children.
Currently there isn't a way to tell the parent process to reload the schema so when a new child is deployed the parent service needs to be restarted to reload the schema.
We understand there there are 1 or more instances of Google Cloud Run running and have ideas on dealing with this, but are wondering if there is a way to restart the parent process at all. Without a way to achieve it, one or more is irrelevant for now. The only way found it by deploying the parent which seems like overkill.
The containers running in google cloud are Alpine Linux with Nodejs, running an express application/middleware. I can stop the node application running but not restart it. If I stop the service Google Cloud Run may still continue to serve traffic to that instances causing errors.
Perhaps I can stop the express service so Google Cloud run will replace that instance? Is this a possibility? Is there a graceful way to do it so it tries to complete and current requests first (not simply kill express)?
Looking for any approaches to force Google Cloud Run to restart or start new instances. Thoughts?
Your design seems, at high level, be a cache system: The parent service get the data from the child service and cache the data.
Therefore, you have all the difficulties of cache management, especially cache invalidation. There is no easy solution for that, but my recommendation will be to use memorystore where all child service publish the latest version number of their schema (at container startup for example). Then, the parent service checks (at each requests, for example) the status in memory store (single digit ms latency) if a new version is available of not. If a new, request the child service, and update the parent service schema cache.
If applicable, you can also set a TTL on your cache and reload it every minute for example.
EDIT 1
If I focus only on Cloud Run, you can in only one condition, restart your container without deploying a new version: set the max-instance param to 1, and implement an exit endpoint (simply do os.exit() or similar in your code)
Ok, you loose all the scale up capacity, but it's the only case where, with a special exit endpoint, you can exit the container and force Cloud Run to reload it at the next request.
If you have more than 1 instance, you won't be able to restart all the running instances but only this one which handle the "exit" request.
Therefore, the only one solution is to deploy a new revision (simply deploy, without code/config change)
I am a node.js developer. I use Amazon ec2 to deploy my node.js apps.
I want to have my node.js service running permanently - restarted if it fails for any reason.
I came across 2 tools . Forever and Upstart
Is there any advantages of using one over the other ?
Is there any other tool which is better ?
Upstart is a system service controller, similar to SysV Init and will start/stop/restart essentially any service registered for it, Node.js-based or not, and it will also automatically start services on system start for you. But Upstart is essentially specific to Ubuntu, and Upstart-specific services won't run on other Linux distros.
Upstart has a SysV Init compatibility layer that you could target,instead, to maintain as broad of a compatibility layer as possible.
Forever is a Node.js application that monitors and restarts other Node.js applications as needed, and as defined by its configuration JSON. Lots of options and fine-grained control over your service without the effort that would be needed to duplicate it in a custom SysV Init script. However, Forever isn't a system service, so if the server is restarted, you'll have to manually start your forever scripts again.
Beyond that, if all you need is something that will restart your script if/when it crashes, and you don't care about it starting automatically on system start, all you need is a bash script as simple as:
#!/bin/bash
while true
do
node ./myScript.js
done
Just to correct a misleading statement in the accepted answer...it is not true that upstart is an Ubuntu-only technology. See:
https://serverfault.com/questions/291546/centos-6-and-upstart
http://searchenterpriselinux.techtarget.com/tip/RHEL-6-ditches-System-V-init-for-Upstart-What-Linux-admins-need-to-know
http://en.wikipedia.org/wiki/Upstart#Adoption
With that, I think it is a much more compelling solution.
My application is built with three distincts servers: each one of them serves a different purpose and they must stay separated (at least, for using more than one core). As an example (this is not the real thing) you could think about this set up as one server managing user authentication, another one serving as the game engine, another one as a pubsub server. Logically the "application" is only one and clients connect to one or another server depending on their specific need.
Now I'm trying to figure out the best way to run a setup like this in a production environment.
The simplest way could be to have a bash script that would run each server in background one after the other. One problem with this approach would be that in the case I need to restart the "application", I should have saved each server's pid and kill each one.
Another way would be to use a node process that would run each servers as its own child (using child_process.spawn). Node spawning nodes. Is that stupid for some reason? This way I'd have a single process to kill when I need to stop/restart the whole application.
What do you think?
If you're on Linux or another *nix OS, you might try writing an init script that start/stop/restart your application. here's an example.
Use specific tools for process monitoring. Monit for example can monitor your processes by their pid and restart them whenever they die, and you can manually restart each process with the monit-cmd or with their web-gui.
So in your example you would create 3 independent processes and tell monit to monitor each of them.
I ended up creating a wrapper/supervisor script in Node that uses child_process.spawn to execute all three processes.
it pipes each process stdout/stderr to the its stdout/stderr
it intercepts errors of each process, logs them, then exit (as it were its fault)
It then forks and daemonize itself
I can stop the whole thing using the start/stop paradigm.
Now that I have a robust daemon, I can create a unix script to start/stop it on boot/shutdown as usual (as #Levi says)
See also my other (related) Q: NodeJS: will this code run multi-core or not?
I am trying to set up a development environment for node.js. I assumed at first that it requires something similar to the traditional, "localhost" server approach. But I found myself at a loss. I managed to start a node.js hello world app from the terminal. Which doesn't looked like a big deal - having to start an app from the console isn't that hard. But, after some tweaking, I found out that the changes aren't shown in the browser immediately - you need to "node [appName here]" it again to run.
So, my question is:
Is there a software or a tutorial on how to create a more "traditional" development server on your local machine? Along with port listening setup, various configurations, root directories etc (things that are regular in stacks like XAMMP, BitNami or even the prepackaged Ubuntu LAMP). Since I'm new at node.js, I can't really be sure I'm even searching for the right things on google.
Thanks.
Take a look at :
https://github.com/remy/nodemon
it'll allow you to do - nodemon app.js
and the server will restart automatically in case of failure.
To do this I built a relatively small tool in NodeJS that allows me to start/stop/restart a NodeJS child process (which contains the actual server) and see/change configuration option and builds/versions of the application, with admin options available on a different tcp port. It also monitors said child process to automatically respawn it if there was a error (and after x failed attempts stops trying and contacts me).
While I'm prohibited from sharing source code, this requires the (built-in) child_process module, which has a spawn method that returns a child process I guess, which contains a pid (process id) which you can use with the kill method to kill said child process. Instead of killing it you could also work with SIGINT an catch it within your child application to first clean up some stuff and then exit. It's relatively easy to do.
Some nice reading material regarding this.
http://nodejs.org/docs/v0.5.5/api/child_processes.html