I'm having a slightly weird, repeatable, but unexplainable problem with screen.
I'm using ansible/vagrant to build a consistent dev environment for my company, and as a slightly showy finishing touch it starts the dev server running in a screen session so the frontend devs don't need to bother logging in and manually starting the process, but backend devs can log in and take control.
However, one of the systems - despite being built from scratch - ends up with an immediately dead screen (it doesn't log anything to screenlog). Running the command manually works fine.
(the command being)
screen -L -d -m bash -c /home/vagrant/run_screen_server.sh
I've even gone to the point of nuking everything vagrant/virtualbox related on the system, making sure it's installing a clean, nightly box. Exactly the same source box works all the other machines.
Are there any other debugging steps I can be taking or is there something I'm missing?
I'm right now trying to do the same with my setup and hit the same problem.
Further testing has shown, that sleep 1 right after calling the screen helped. It seems the ssh script that ansible calls exits before the screen call is fully detached (or something else, that would explain that the sleep 1 helps)
I've also found Can't get Fabric's detached screen session example to work with the same suggestion.
Related
There are a lot of pieces so I don't expect anyone to be able to answer this without seeing every configuration. But maybe people can tell me how to look for diagnostics or kind of how the major pieces fit together so that I can understand what I'm doing wrong.
I have a Tencent CVM instance running Ubuntu Server.
I also have a domain name pointing to the ip address of that server.
I start an nginx service to listen to port 1337 and pass requests to example.com/parse
I have mongodb running inside of a docker container, listening on port 27017.
Inside of index.js, I have the databaseURI set as 'mongodb://localhost:27017/dev' and the SERVER_URL set as 'https://example.com/parse'
When it's time to deploy the Parse Server instance, I use screen inside of my current ssh session, run npm start, and then detach the screen, and then kill my ssh session by closing the terminal.
Finally, I run the parse dashboard on my local machine with serverURL 'https://example.com/parse'
And everything works great. I add items to the database via the test page that comes with the Parse Server repo. I add items to the database via cloudcode calls from Python. I add and delete classes and objects via the dashboard. The behavior is exactly like I should expect.
And it continues that way for anywhere between 12-72 hours.
But after a few days of normal operation, it will happen that I open parse dashboard and everything is gone. I can start adding things again and everything works right, but nothing persists for more than 72 hours.
There's a lot I don't understand about the anatomy of this, so I figured maybe using screen and then detaching and closing the terminal causes some process to get killed and that's my mistake. But when I run top, I can see everything. I can see npm start is running. I can see mongo is running. When I docker ps to check the container, it's still there doing fine. The nginx service is still running.
Can anyone suggest a way for me to start diagnosing what the problem is? I feel like it's not any of the configs because if that was the problem, it wouldn't work fine for so long. I feel like it must be how I'm deploying it is causing something to reset or causing some process that's supposed to be always running to die.
Edit: For posterity I'll summarize the below as a solution in case you've come here struggling with the same issue. #Joe pointed me to db.setProfilingLevel(), level 2 with the slowms=0 option for max verbosity. Those logs are written to the file indicated within mongodb.conf. Docker doesn't persist storage by default, so you should have a named volume. The syntax is $docker volume create <volume_name>. And the syntax to attach the volume when you create the container is to add a v flag like -v <volume_name>:. And finally, I was running mongodb in a container because that's the workflow I saw in tutorials. But it's solving a problem I didn't have and it was simpler to start mongodb as a service without a container.
I just installed all necessary dependencies on a new AWS EC2 Linux 2 server and launch my long_running_script.py. This script essentially performs a few operations then sleeps for a few hours on a never ending loop.
When I launched the script initially, I saw the correct output and all was fine. I disconnected from the instance, and when I reconnected I expected to see the same output as before.
Instead I can't seem to see any script output or see it running after typing in the 'ps aux' command.
Did disconnecting from the instance somehow abort the script? If so, how can I make sure it stays running?
Appreciate your help.
Did disconnecting from the instance somehow abort the script? If so, how can I make sure it stays running?
Yes it did. There are many ways to solve this. You can launch it using tmux or screen. Lunching your program in these "shells" will keep it running after you log out.
There is also nohup and pm2 which could also be helpful.
I don't have Sudo access, so currently i can't install 'Forever' https://www.npmjs.com/package/forever
Instead i am simply using 'Screen'.
I am running a node.js server, at a random point, the node server stops, and screen exits. I cannot seem to collect any error data on this. I seem to be completely unaware of why its happening and cannot think of a way to catch what is happening. It doesn't happen often (maybe 1 time per day). When i load putty back up and login to my Apache server through terminal, i type screen -x or screen -r and it tells me there are no screens attached. The node server process definitely stops because the app it runs stops working.
Obviously i can't post all the code here, there is tons of it. But everything appears to work wonderfully, except every now and then, something goes wrong and it closes the attached screen.
If there was a problem with the node server, i would expect a crash, and the attached screen would stay attached. There would be an error outputted to the terminal for me to see when i open it. But in this case, it totally closes the attached screen.
Does anybody know what kind of error can cause this?
On a side note, is there an alternative to 'Forever' that can be installed without Sudo access?
My node version wasn't correct which is why Forever wasn't installing. I didn't need SUDO after all. I am now using Forever and hopefully this will shed light on what is going on as i have a out.log file which should catch whatever the problem is. :-)
I'm really new to node.js. My friend helped me set up a node app to run a java process I need running on a server at all times. It works perfectly, except the only way I can see the standard i/o is if I use node app.js. I've looked into both forever and pm2, however neither of these use standard i/o, which I really need for this server to run commands. Could somebody help me out please? Thanks!
Assuming you have a *nix-based server:
You could use GNU Screen.
Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.
In plain words, you would have access to always-running processes on server and their input-output from your local command line.
After logging in to your server, all you need to do is this:
Start new screen screen -S <name>
Run you java process
Detach from screen screen -d <name>
That's it! Your java process keeps running and you could interact with it by reattaching to the screen session like this: screen -r <name>
Useful Link: GNU Screen Quick Reference
Even cooler would be creating your own service using an Upstart script, which you could then call directly from your local machine with:
Create your own service using Upstart script.
I'm running HSQLDB in server mode on a Linux server and finding that it occasionally gets killed. I'd like to be able to detect that it's stopped running and then kick off a process that starts it up again.
The DB isn't running very often, so polling would have to be very frequent--once every five minutes.
Look at Monit:
Monit is a free open source utility for managing and monitoring, processes, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.
If you are using soem type of Debian, you might try installing HSQLDB using "apt-get install hsqldb-server. That will give you a nice install and the ability to start with "/etc/init.d/hsqldb-server start"
This will also take care of restarting it if your machine reboots. If you get everything installed correctly the problem of it getting killed may just go away.
I was running into some weird issues starting and stopping hsqldb, but once I got it installed correctly everything took care of itself.