I have created a Systemd unit file to run Mongodb on system startup. I have also created a systemd service to run my node application on startup after mongodb is started. The service for mongodb works fine, but for some reason my service for the Node application tries to run and then gives the error: "MongoError: failed to connect to server [localhost:27017] on first connect". If I start the mongodb service using $systemctl start mongodb and then start my Node application using $/usr/bin/node /node_app_slot/server.js It seems to work fine. So the problem seems to be with my systemd unit file for my Node server.
I used this for the mongodb systemd service https://gist.github.com/jwilm/5842956
And here is my node_server.service:
[Unit]
Wants=network.target mongodb.service
After=network.target mongodb.service
[Service]
ExecStart=/usr/bin/node /node_app_slot/server.js
[Install]
WantedBy=multi-user.target
Im doing this on an intel edison set up in Access Point mode using hostapd. The OS is Yocto and is up to date with the latest release.
I can't see where Im going wrong. I will really appreciate it if someone could guide me in the right direction!
Thanks!
Your systemd syntax is correct, although you may want to use the network-online.target instead of network.target.
A workaround to consider is to add a 5 second sleep before your MongoDB app starts. Since this is happening at server boot time, the extra 5 seconds are unlikely to make a practical difference but may solve your problem.
ExecStartPre=/bin/sleep 5
It is curious that you are copy/pasting a MongoDB systemd service file from the internet when MongoDB ships with their own systemd service files. You didn't mention your OS version, systemd version or MongoDB version, but I would still recommend referencing the official MongoDB systemd configuration files if aren't using a version of MongoDB that ships with the files.
I had similar problem. The problem may be the fact that you try to connect before the network service has finished starting. You can take a look at here to see if it solves your problem:
https://unix.stackexchange.com/questions/126009/cause-a-script-to-execute-after-networking-has-started
Related
I have a website with Nginx installed as a reserve proxy for an ExpressJS server (proxies to port 3001). This uses Node and ReactJS for my frontend application.
This is simply a testing website currently, and isn't known or used by any users. I have this installed on a Digital Ocean Droplet with Ubuntu.
Every morning when I wake up, I load my website and see 502 Bad Gateway. The problem is, I don't know how to find out how this happened. I have PM2 installed which should automatically restart my ExpressJS server but it hasn't done so, and when I run pm2 list, my application is still showing online:
When I run pm2 logs, I get the following error (I am running this as an Administrator):
So I'll run pm2 restart all to restart the app, but then I don't see any crash information. However on this occasion when taking this screenshot, there were a couple of unusual requests. /robots.txt, /sitemap.xml and /.well-known/security.txt, but nothing indicating a crash:
When I look at my Nginx error.log file, all I can see is the following:
There is, however, something obscure within my access.log ([09/Oct/2018:06:33:19 +0000]) but I have no idea what this means:
If I run curl localhost:3001 whilst the server is offline, I will receive a connection error message. This works fine after I run pm2 restart all.
I'm completely stuck with this and even the smallest bit of help would be appreciated greatly, even if it's just to tell me I'm barking up the wrong tree completely and need to look elsewhere - thank you.
I think you should check this github thread, it seems like it could help you.
Basically, after few hours, a Nodejs server stop functioning, and the poor nginx can not forward its requests, as the service listening to the forward port is dead. So it triggers a 502 error.
It was all due to a memory leak, that leads to a massive garbage collection, then to the server to crash. Check your memory consumption, you could have some surprises. And try to debug your app code, a piece (dependency) at the time.
Updated answer:
So, i will add another branch to my question as it seems it has not helped you so far.
You could try to get rid of pm2, and use systemd to manage your app life cycle.
Create a service file
sudo vim /lib/systemd/system/appname.service
this is a simple file i used myself for a random ExpressJS app:
[Unit]
Description=YourApp Site Server
[Service]
ExecStart=/home/user/appname/index.js
Restart=always
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/home/user/appname
[Install]
WantedBy=multi-user.target
Note that it will try to restart if it fails somehow Restart=always
Manage it with systemd
Register the new service with:
sudo systemctl daemon-reload
Now start your app from systemd with:
sudo systemctl start appname
from now on you should be able to manage your app life cycle with the usual systemd commands.
You could add stdout and stderr to syslog to understand what you app is doing
StandardOutput=syslog
StandardError=syslog
Hope it helps more
You cannot say when exactly NodeJS will crash, or will do big GC, or will stun for other reason.
Easiest way to cover such issues is to do health check and restart an app. This should not be an issue when working with cluster.
Please look at health check module implementation, you may try to use it, or write some simple shell script to do the check
I'm having a problem with PostgreSQL 9.5+173 on Ubuntu 16.04 and I happened to stumble across the following threads in my research that somewhat describes the behavior I'm seeing:
https://www.postgresql.org/message-id/CAFyxdeT%2B%3Dx-d0oNbFPoe%2B4xnt0Qdfi%2BzAEn%2BrQmEK0AZbJFRtg%40mail.gmail.com
https://www.postgresql.org/message-id/562E4453.5090803%40aklaver.com
Long story short I have a fresh install of Ubuntu 16 with nothing on it and PostgreSQL running. I've stopped PostgreSQL changed the data directory and port and a couple other settings and it starts back up fine.
I can start and stop PostgreSQL manually via systemctl without any problems. I can also connect to the database and can verify that it is running via a ps ax | grep postgres.
However, after I reboot PostgreSQL will not start up. Any attempt to start it up via systemctl start postgresql.service doesn't do anything and does not fail. The only way I am able to get it started is if I call systemctl start postgresql#9.5-main.service.
I did some investigation and looked at both the postgresql.service and postgresql#9.5-main.service scripts and realized that the postgresql.service script does nothing as stated in the thread above and that the postgresql#9.5-main.service has the PartOf directive which means it should be getting triggered from the postgresql.service as the sytemd docs state, but it isn't for some reason. Basically I'm at a loss as to why everything works before I reboot and then doesn't work after I reboot. Is there something I'm missing? I'm starting to go CRAZY over something so simple.
Update: I added an ExecStartPre=/bin/touch /tmp/postgresq.log to the postgresql#9.5-main.service to see if it's actually getting called on boot and it is not. Manually calling systemctl start postgresql#9.5-main.service creates the file in the /tmp directory.
Update: I have also found that calling systemd daemon-reload after reboot will allow me to start postgres via the systemctl start postgresql command.
Did you try doing systemctl enable postgresql? This will tell systemd to start this service after boot. Try rebooting after that.
Turns out that the problem was the fact that I symlinked /etc/postgresql/9.5/main/ across partitions to a custom partition that wasn't available right away, so when PostgreSQL tried to start on boot it couldn't because it's configuration files were not available. This describes what was happening since I could start PostgreSQL manually after I logged in.
I'm trying to get a nodejs server to run on startup, so I created the following systemd unit file:
[Unit]
Description=TI SensorTag Communicator
After=network.target
[Service]
ExecStart=/usr/bin/node /home/pi/sensortag-comm/sensortag.js
User=root
[Install]
WantedBy=multi-user.target
I'm not sure what I'm doing wrong here. It seems to fail before the nodejs script even starts, as no logging occurs. My script is dependent on mysql 5.5 (I think this is where I'm running into an issue). Any insight, or even a different solution would be appreciated.
Also, it runs fine once I'm logged into the system.
Update
The service is enabled, and is logging through journalctl. I'll update with the results on 7/11/16.
Not sure why it didn't work the first time, but upon checking journalctl the issue was 100% that MySQL hadn't started. I once again changed it to After=MySQL.service and it worked perfectly!
If there is no mention of the service at all in the output of journalctl that could indicate that the service was not enabled to start at boot.
Make you run systemctl enable my-unit-name before your next boot test.
Also, since you depend on MySQL being up and running, you should declare that with something like: After=mysql.service. The exact service name may depend on your Linux distribution, which you didn't state.
Adding User=root adds nothing, as system units would be run by root by default anyway.
When you said "it fails", you didn't specify whether it was failing at boot time, or with a test run by systemctl start my-unit-name.
After attempting to start a service, there should be logging if you run journalctl -u my-unit.name.service.
You might also consider adding StandardOutput=journal to your unit file to make sure you capture output from the service you are running as well.
my group and I are running a server that is based upon Django and uses mod_wsgi to run an Apache server. We will not be working on this project after it is over, so I am attempting to set up cronjob similar functionality to check if the apache server has shut down(system restart or power failure), and if it has, will restart the server for me. I've found documentation on how to check if an apache server is down and restart the server if it is, but our server uses https and thus our start command is pretty verbose.
Can I simply use the functionality provided in these examples:
https://askubuntu.com/questions/277389/cron-job-to-restart-apache
https://www.digitalocean.com/community/tutorials/how-to-use-a-simple-bash-script-to-restart-server-programs
Or do I need a much more complicated process to make this happen?
The command we use to initially start the server is
python manage.py runmodwsgi --host 0.0.0.0 --port 8001 --https-port 8000 --ssl-certificate (certificate Location) --server-name (Domain Name)
I'm pretty new to Linux and using both Mod-wsgi as well as Apache so any help is greatly appreciated.
I suppose it is not good way to resolve this problem.
I recommend you use monit (https://mmonit.com/). It is cool program for checking services.
apt-get install monit
Apache restart configuration directives:
check process httpd with pidfile /var/run/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout
You are better off using the --setup-only option to mod_wsgi-express or the Django integration for it, to generate the configuration but not run it. Then as others have mentioned, integrate it into the system service manager.
The two commands for starting and stopping the Apache/mod_wsgi instance would be apachectl start and apachectl stop, where apachectl is that which was generated when running with the additional --setup-only option.
When running it as a system service, also make sure you use the --server-root option to specify a more persistent location for the generated configuration. Do not use the default under /tmp if running for anything but temporary development sessions as some Linux systems will remove files under /tmp causing things to start failing after a while.
Also, since under a service manager it would generally be starting as root, particularly if listening on port 80 is a requirement, ensure you use the --user and --group options to specify what user/group your Python web application should run as.
Read:
https://pypi.python.org/pypi/mod_wsgi
for more details of the --setup-only option and start-server commands for generating the configuration. Because you are using the Django integration, you will need to use the --setup-only option.
For more informed helped, bring your issue to the mod_wsgi mailing list. The mod_wsgi-express way of running Apache/mod_wsgi is new enough that unlikely that anyone here is really going to know much about it.
There is no need to do this at all. There is no reason to start up Apache manually; once it's installed as a system service, Ubuntu will start it up automatically on restart or crash.
You should reflect on why you feel the need to do this for Apache specifically, and not any of the other system services you depend on, such as the database.
I have a simple Meteor web app that I am trying to host on my own server.
Server details;
Debian (Jessie) Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux
systemd
NodeJS version 0.10.36
Meteor version 1.0.3.1
MongoDB version 2.6.7
/etc/systemd/system/customwebapp.service file;
[Service]
ExecStart=/usr/bin/node /opt/customwebapp/bundle/main.js
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=customwebapp
User=customwebapp
Group=customwebapp
Environment=NODE_ENV=production MONGO_URL='mongodb://localhost'
[Install]
WantedBy=multi-user.target
I create the service account using;
sudo useradd -mrU customwebapp
I build the Meteor app using;
sudo meteor build --directory /opt/customwebapp
cd /opt/customwebapp/programs/server
sudo npm install
I try to register the service with;
sudo systemctl enable customwebapp
sudo systemctl start customwebapp
Syslog reports the following;
/opt/customwebapp/bundle/programs/server/node_modules/fibers/future.js:173
Error: URL must be in the format mongodb://user:pass#host:port/dbname
at Error (<anonymous>)
I have tried for hours to fix this problem.
I have used a number of different MONGO_URL values.
I created an admin or root user in MongoDB using the mongo command and put the user:pass#localhost:27017/customwebapp format into the URL environment variable.
No matter what I do it comes back with the same error.
I have also tried using Meteor Up but it seems to configure Upstart not systemd.
And it goes without saying, I have tried a large number of solutions from SO. Not all of the solutions I have tried are listed here.
I suspect the reason I am having issues is because of systemd.
This is my first question on SO because, to be honest, most problems I have somebody else has had the issue before me. SO is brilliant for helping me and has many times. Now I'm hoping SO will help again.
Thanks for taking the time to help.
OK, I found the issue after digging into source code in the NodeJS core module called Url
As you can see from my question above I was setting the Environment in the service unit file using this line;
Environment=NODE_ENV=production MONGO_URL='mongodb://localhost'
The Node Url module was detecting the single quotes ' and replacing them with %27. This caused the Url.parse function to fail setting the host value.
Here is the correct way to set the Environment values on multiple lines and without the single quotes;
Environment=NODE_ENV=production
Environment=MONGO_URL=mongodb://localhost
Environment=ROOT_URL=http://localhost:8080
Environment=PORT=8080
It's always the little syntax issues isn't it.