Determining Website Crash Time on Linux Server - linux

2.5 months ago, I was running a website on a Linux server to do a user study on 3 variations of a tool. All 3 variations ran on the same website. While I was conducting my user study, the website (i.e., process hosting the website) crashed. In my sleep-deprived state, I unfortunately did not record when the crash happened. However, I now need to know a) when the crash happened, and b) for how long the website was down until I brought it back up. I only have a rough timeframe for when the crash happened and for long it was down, but I need to pinpoint this information as precisely as possible to do some time-on-task analyses with my user study data.
The server runs Linux 16.04.4 LTS (GNU/Linux 4.4.0-165-generic x86_64) and has been minimally set up to run our website. As such, it is unlikely that any utilities aside from those that came with the OS have been installed. Similarly, no additional setup has likely been done. For example, I tried looking at a history of commands used in hopes that HISTTIMEFORMAT was previously set so that I could see timestamps. This ended up not being the case; while I can now see timestamps for commands, setting HISTTIMEFORMAT is not retroactive, meaning I can't get accurate timestamps for the commands I ran 2.5 months ago. That all being said, if you have an idea that you think might work, I'm willing to try (as long as it doesn't break our server)!
It is also worth mentioning that I currently do not know if it's possible to see a remote desktop or something of the like; I've been just ssh'ing in and use the terminal to interact with the server.
I've been bouncing ideas off with friends and colleagues, and we all feel that there must be SOMETHING we could use to pinpoint when the server went down (e.g., network activity logs showing spikes around the time that the user study began as well as when the website was revived, a log of previous/no longer running processes, etc.). Unfortunately, none of us know about Linux logs or commands to really dig deep into this very specific issue.
In summary:
I need a timestamp for either when the website crashed or when it was revived. It would be nice to have both (or otherwise determine for how long the website was down for), but this is not completely necessary
I'm guessing only a "native" Linux command will be useful since nothing new/special has been installed on our server. Otherwise, any additional command/tool/utility will have to be retroactive.
It may or may not be possible to get a remote desktop working with the server (e.g., to use some tool that has a GUI you interact with to help get some information)
Myself and my colleagues have that sense of "there must be SOMETHING we could use" between various logs or system information, such at network activity, process start times, etc., but none of us know enough about Linux to do deep digging without some help
Any ideas for what I can try to help figure out at least when the website crashed (if not also for how long it was down)?

A friend of mine pointed me to the journalctl command, which apparently maintains timestamps of past commands separately from HISTTIMEFORMAT and keeps logs that for me went as far back as October 7. It contained enough information for me to determine both when I revived my Node js server as well as when my Node js server initially went down

Related

Cygwin intermittently loses it's mapped drives in /cygdrive

So, I have a collection of Windows Server 2016 virtual machines that are used to run some tests in pairs. To perform these tests, I copy a selection of scripts and files from the network on to the machine, before performing the tests.
I'm basically using a selection of scripts that have existed around here since before my time and whilst i would like to use other methods, so much of our infrastructure relies on these scripts that overhauling the system would be a colossal task.
First up, i sort out the mapped drives with
net use X: \\network\location1 /user:domain\user password
net use Y: \\network\location2 /user:domain\user password
and so on
Soon after, i use rsync to copy files from a location in /cygdrive/y/somewhere to /cygdrive/c/somewhere_else
During the rsync, i will get errors that "files have vanished" (I'm currently unable to post the exact error, I will edit this later to include this). When i check what's currently in the /cygdrive directory, all i see is /cygdrive/c and everything else has disappeared.
I've tried making a symbolic link to /cygdrive/y in a different location, I've tried including persistent:yes on the net use command, I've changed the power settings on the network card to not sleep. None of these work.
I'm currently looking into the settings for the virtual machines themselves at this point, but I have some doubts as we have other virtual windows machines that do not seem to have this issue.
Has anyone has heard of anything similar and/or knows of a decent method to troubleshoot this?
Right, so I've been working on this all day and finally noticed a positive change, but since my systems are in VMware's vCloud, this may not work for some people. It's was simply a matter of having the VM turned off and upgrading the Virtual Hardware Version to the latest version. I have noticed with this though, that upon a restart, one of the first messages that comes up mentions that the computer is "disabling group policies".
I did a bit of research into this and found out that Windows 8 and 10 (no mention of any Windows Server machines) both automatically update Group Policies in the background, disconnecting and reconnecting mapped drives to recreate them.
It's possible that changing the Group Policy drive from "recreate" to "update" should fix this issue, and that the Virtual Hardware update happened to resolve this in a similar manner.

What do unrecognized GET requests mean?

It's a new Amazon EC2 instance, it's live maybe a couple of hours a day, I've just installed NodeJS and still playing with it. And then I get this in my Putty SSH window:
I don't know what those two last requests mean. I don't have robots.txt and I definitely don't have any HTML file whatsoever (it's all in Jade).
Do I havet to be concerned?
Its probably a bot. Generally you don't need to worry about it, unless they are generating to many requests to the point that it causes performance issues with your application.

How to deploy a nodejs application to google cloud?

I'm trying to deploy a nodejs application to google cloud, I've posted several SO questions about the errors I've been getting and how to fix them. Let me illustrate my endeavours...
problem 1) I tried following the tutorial, the hello world app deployment. When I entered the command to deploy,
$ gcloud preview app deploy app.yaml --set-default
It gave me an error saying managed vms aren't available for your application's region. I contacted someone from Google, a technical support representative from google, said his name was Ling, and he said he "wasn't aware managed vms weren't supported in India" and he wanted me to point out where I found that information. So I sent him this, note one of the answers. OK.....
problem 2) So, I was stuck, didn't really know what to do. Searched "how to deploy nodejs app on google cloud" and found a video in which a google developer informed me that there are "two main ways to run your application in the cloud, infrastructure as a service and platform as a service". Running it on app engine was the platform as a service, and that was off the list so I went with my only remaining option. There is a tutorial, the bookshelf app tutorial, also by google, and the last part describes how to run your application on the compute engine. I tried to go through the tutorial, but when I entered the command
git push cloud 7-gce:master
to push my code to the repository, it's asking me for my username and password, It never mentioned that in the tutorial, I don't know what to enter. I found this question but I'm not sure it applies to my situation. So next I went through some tutorials suggested by Zachary Newman, and I wasn't able to follow the tutorial where they use the ubuntu because when they tell you to hello.js file, it's really weird because I had to ssh into the ubuntu image vm I created and everything was really slow, I'm not familiar with commands on linux so I had to look up how to "save and exit" after creating the hello.js file. I got the "web server" to run, it's the reverse proxy nginx server, but since I don't know what the APP_PRIVATE_IP_ADDRESS is, when I accessed the page, I could see the nginx page but not the "hello world" message that's supposed to display. Please see the Prerequisites heading to see what the tutorial was trying to teach me. OK....
problem 3) On the google developers console, once you log in, there is a navigation section on the left, there is a link to the "Cloud Launcher". In there I found a link to create a Virtual Machine with nodejs installed, it's created by Bitnami. I clicked it and my vm was running, I went to the external ip address provided, and saw a Congratulations (ugh...) page by Bitnami. I didn't really know what to do from there, so I posted another SO question. A bitnami developer told me I "need to copy your files to the remote repository, create the configuration files and after that you have to include that configuration in the configuration of Apache to serve the application". How?!?! He just posted a link as well to a bitnami wiki page. I used filezilla to connect to my vm, and I tried to upload my application files to the vm. I was able to upload everything except the node modules (they were taking too long to upload and for some reason the ftp client kept loosing the connection with the server after 10 minutes or so of uploading). So I thought I would just ssh into the instance and try to npm install my dependencies. But it gave me more errors...
problem 4) I don't know why this post was put on hold, but I found this. I can't find the exact page right now but I read somewhere in the google docs that we can't ssh into an instance as root/administrator by design, and that we have to perform some advanced configuration in order to be able to do this. THE PROBLEM is that I can't install the dependencies unless I ssh in as an administrator....:'( So I just uploaded my node modules with filezilla. Everything looks good to go, I ssh into my instance and I cd into my application directory and enter the command
node app.js
and almost as if to taunt me, it console logs out "connected to mongo...." and "Listening on port 3000.....". I get so excited, I think it's working.....
problem 4) The server never really started running I think, it wasn't able to connect to the mongodb database (another instance) that I was running, this was the command to connect to the mongodb instance in my app.js file
mongoose.connect('mongodb://****/myappdatabase');
where the **** indicates the external IP of my mongodb instances. After about 5 minutes, I get an error on the ssh browser
Error: connect ETIMEDOUT ****:27017
BUT WHO HAS TIME TO READ ALL OF THIS RIGHT?
I'm not a developer, I don't have much experience with programming (close to zero actually). I'm a hobbyist. I picked up a book on c programming by stephen g. kochan about 5 or 6 months ago, and since then, I've just been following tutorials on the internet, a programming in java book, and some reference material. I wanted to create a website.
I learned some html, css, javascript and jquery off of w3schools and went through some tutorials on creating a basic node server and serving some html pages, that's literally all my application does. That's all I want it to do.
I used the express framework for node, and mongoose to connect to my mongodb running locally. Socket.io for a little bit of networking. I created the website for fun and I would really like to host it and share it with my friends, I made it this far, it would be really embarrassing to quit now because I can't host the website. I went through a lot more than what I posted on this question by the way.
Now I would really appreciate some help, even some guidance as to what I need to know, maybe books I should read or documentation I should go through to become more familiar with website hosting would be great. I am going to offer this question up for a bounty in 2 days but until then, if there is anyone that went through this and would like to help for no personal gain of their own, your saving my life. I've been going at this for about 4 days with no luck. Thanks and let me know if there is anything you would like to know about my application. I mean seriously, how hard could it be, I'm doing something (or everything) terribly wrong.
Wow dude. I thought about flagging this, but you appear to genuinely want to learn to program and need help. And experienced coders teaching novice coders is what this site is all about. So, here we go.
First and foremost, Linux is your friend. Learn it. Use it. I don’t use Ubuntu, but you should. Use 14.04 because it’s a forgiving disto and comes cram-packed with things you need when you’re just learning.
Time to address your questions:
Problem 1: you are using a gcloud preview product. Those are awesome, but not for beginners because you don’t know what you’re doing. Just stand up an Ubuntu instance on an n1-standard and rock it. No bells. No whistles. Learn Ubuntu linux with the same passion that you taught yourself how to program a web site.
Problem 2: Don’t use git. I use git every day and it’s awesome, but you have a long way to go and git is hella confusing. Take your workspace, tar/gzip it and just ssh the whole shebang to cloud. Then unzip it and you’ll have your whole code right there on your server. Caveat: never do this professionally. But for someone just learning … small bites dude … small bites.
Problem 3: Don’t use Bitnami pre-built images. It robs you of the opportunity learning how to do this from scratch and that is an invaluable experience. Here are two commands you must learn:
apt-get update
apt-get install <insert thing you want to install>
That’s it. Not rocket science and you will discover the power of package management on linux.
Problem 4: You are over-thinking this. For real, learn apt-get. It will get you so far, you’ll curse yourself for not using it earlier. apt-get node. apt-get mongo. And slam it all on one instance. Is that what I do professionally? Hell no. But it’s all about baby steps when you combine dev-ops with programming and you need to take those steps.
Errata: You wanted someone to recommend a book or a website or guidance. All you need to know is already on Ubuntu. They are called man pages. It’s short for manual. You can buy all the books in the world off Amazon and watch infinite youtube pages. It won’t compare to simply maning commands. You gotta trust me on this.
In summary: Just create an Ubuntu instance. tar and gzip all your code and scp it to your instance. Learn apt-get and for real, you are set. You will totally destroy your instance many times. So just delete it, re-create it, and keep starting over. Each time, you will be wiser and smarter than the time before. I tutor novice developers all the time. You gotta start small and work your way up. Eventually you’ll understand why you’d want to use git. Why you want to scale. Why LXC is amazing. Why GCE’s managed instance and VMs are crazy baller, and why scripting your deployment is a best practice. But first you have to understand how to build an instance and get your code there in any way possible. You’ll grow from there. Good luck, buddy.

oracle temporary ora 12505 error after linux startup

I am experiencing a very strange behavior with oracle, maybe somebody can help me, let me summarize it real quick:
My OS of choice is debian linux, I am using Oracle XE 11.0.2.0. On linux startup, I run a script file which is located under /etc/init.d/. I added the following line to make oracle start on system start:
/etc/init.d/oracle-xe start
Right after this line , I run my application from the script, my application heavily relies on the oracle db, therefore once oracle starts, I am positive that my application will run ok. Unfortunately my assumption seems wrong.Here's why: I set up similar set up in 3 machines, in 2 of them I see weird behavior, after system start oracle db is not responding to connection requests, Even though oracle-xe start command completed executing.
My observation is the following, if I run my application right after oracle-xe start is executed, I receive ora-12505 errors at least for a minute: "TNS listener does not currently know of SID" . After a minute everything stabilizes, and my application starts working ok. 1 minute without a db on system startup is not acceptable for me performance-wise, therefore I am trying to solve this problem.
Surprisingly it does not happen in one of the other linux boxes I have here, I am not quite sure what is different on that box. I compared ora files, but couldn't find any difference, it seems like a wild goose chase...
I would be so grateful if anybody has experienced and solved ths problem before and shares that valuable solution with me.
I think I found the problem, looks like I am starting oracle-xe instance before I assign network interfaces an IP address, in that case it takes some time for oracle to receive connections, that requires me to set static ip on the linux boxes, which is something I don't want. Is there a solution so that I can still assign IP addresses later on?

Linux: What should I use to run terminal programs based on a calendar system?

Sorry about the really ambiguous question, I really have no idea how to word it though hopefully I can give you more detail here.
I am developing a project where a user can log into a website and book a server to run a game for a specific amount of time. When the time is up the server stops running and the players on the server are kicked off. The website part is not a problem, I am doing this in PHP and everything works. It has a calendar system to book a server and can generate config files based on what the user wants.
My question is what should I use to run the specific game server on the linux box with those config files at the correct time? I have got this working with bash scripts and cron, but it seems very un-elegant. It literally uses FTP to connect to the website so it can download all the necessary config files and put them in a folder for that game and time. I was wondering if there was a better way of doing this. Perhaps writing a program in C, but I am not sure how to go about doing this.
(I am not asking for someone to hold my hand and tell me "write this code here", just some ideas of a better way of approaching this problem)
Thanks so much guys!
Edit: The webserver is a totaly different machine. I would theoreticaly like to have more than one game server where each of them "connects" (at the moment FTP) to the webserver, gets a file saying what it has to do at a specific time and downloads any associated files then disconnects.
I think at is better suited for running one time jobs than cron.
For a better approach for the downloading files etc, you should give more details on your setup (like, the website and the game server, are they on the same machine? Or the same network? etc etc.
You need a distributed task scheduler. With that, you can:
Schedule command "X" to be run at a certain time.
Specify the machine (or ask it to pick a machine from a pool of available machines)
Webserver would send request to this scheduler via command line or via web service when user selects a game server and a time.
You can have a look at : http://www.acelet.com/super/SuperWatchdog/index.html
EDIT :
One more option :http://jobscheduler.sourceforge.net/

Resources