I'm trying to install ElasticSearch on a Linux environment. I'm almost there, but I'm doing something wrong.
Basically, my issue is this: when I run bin/elasticsearch in the terminal, I can access http://localhost.com:9200 just fine and run queries. But when I end the SSH session, I get a 404 when I try to hit http://localhost.com:9200 both in my browser window and through CURL.
This is the process I used to install ElasticSearch:
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.0.tar.gz
tar -zxvf elasticsearch-1.1.0.tar.gz
cd elasticsearch-1.1.0
bin/elasticsearch
Like I said, this works, and I can hit http://localhost.com:9200 to run queries, but only as long as the SSH session is active.
Is there something obvious I'm doing wrong? I'm a novice at command line stuff, so I'm trying to feel my way through.
From 1.0 onwards the default option launches the service in foreground
ElasticSearch Documentation http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_system_and_settings.html
Try using the "-d" option which runs the server as a daemon.
Related
The detailed installation guide to install Redis on Mac
Hello Everyone,
I recently stumbled upon a YT video on Redis Crash Course by "BRAD" in Traversy Media Channel(https://www.youtube.com/channel/UC29ju8bIPH5as8OGnQzwJyA). Below are the following that I got stuck in while Installing Redis.
I was unable to download Redis through CLI i.e. wget
https://download.redis.io/releases/redis-6.2.6.tar.gz and note, I
used curl as wget was not functional.
I was unable to start the Redis-Cli and it tortured me with an Error:
Could not connect to Redis at 127.0.0.1:6379:connection refused not
connected> Below are the steps that I followed to install and run
successfully.
[Solution]Problem statement 1:
Instead of downloading through CLI, I tried downloading the "tar.gz" file directly. Downloaded the stable version 6.2.6 and then followed the below CLI commands.
$ tar xzf redis-6.2.6.tar.gz $ cd redis-6.2.6 $ make
This made the job easy to make a binary. Post which, I followed the Redis documentation to run the redis-server. And, it worked fine.
[Solution]Problem statement 2:
As I said, I was unable to run the redis-cli even though, I was able to successfully run the redis-server. I tried several websites and StackOverflow to understand the concept behind the error. That's when I realized the redis-server and redis-client are two separate executable files/process so to make the redis-client work, you should keep in mind that the redis-server should run either in background or in other terminal.
Note, if you're executing the redis-server in the same terminal, then make sure to run the server in the background using the below command.
redis-server --daemonize yes
This should solve the problem, now try using the redis-cli. It will work perfectly.
Now, you can see port 6379 with the localhost IP, make a test PING and confirm it is connected.
I have a script in which I build a mlagents_envs.environments.UnityEnvironment that successfully launches and works when I run the script from terminal sessions started on my ubuntu machine (that has a GUI). And if I ssh into the machine, I can run these scripts from tmux sessions that were originally created locally on my machine. If, however, I try to run the script from a terminal session created through the remote ssh connection, the script hangs when trying to create the UnityEnvironment. It just says:
Found path: <path_to_unity_executable>
and eventually times out.
I've tried to run the script with a virtual display and it still doesn't work. Specifically, I've tried:
$ xvfb-run --auto-servernum --server-args='-screen 1 640x480x24:64' python3 python_script.py -batchmode
$ xvfb-run --auto-servernum --server-args='-screen 1 640x480x24:64' python3 python_script.py
And I've tried the instructions found here: https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-on-Amazon-Web-Service.md
Has anyone encountered this issue? Do you have any suggestions?
The solution ended up being fairly simple. I just needed to specify the right DEVICE before running the script.
$ DEVICE=:1 python3 python_script.py
If anyone else runs into this, you might also need to enable X11 forwarding in both the ssh settings on the server and the client. I'm not 100% sure.
I apologize if this is a stupid questions; I am a complete newbie when it comes to cloud computation.
I am using Google Compute instances to run python scripts with GPU support. Unfortunately, it seems that for the script to run, my computer has to be on and the terminal connecting me to my instance must be open.
I am wondering if there is any way to run python scripts on instances in Google Cloud completely remotely, and just SSH in to see when the script is finished.
I have considered using IPython notebooks or something similar, but that code I am running requires a very specific Anaconda environment, and is meant to be run via terminal.
Edit 1:
The reason I think I need to have the console connecting me to the instance is because I tried to test it out by writing a small script to make files every minute. My process was as follows:
1. Create an instance, SSH in through the google cloud Instances page
2. Create a new python script with this code:
import time
i=0
while 1:
tmp_file = open("tst"+str(i)+".txt","w")
tmp_file.write(str(i))
tmp_file.close()
i += 1
time.sleep(60)
I then ran this code, confirmed it worked by SSHing in with a different console.
I closed the console with the program running in it. After that, files stopped being created.
Ideally, I would like a situation where I could run such a script, close out of the terminal window and have the execution of the script be unassociated with things like whether I have the console open or whether my device is on. I would like to just be able to SSH in and see the result of a script once it is finished.
I'm also a total novice when it comes to GCE and Python, so you're in good company! I had a similar problem when learning to use GCE. I opted to use a start-up script but I am not sure how well this will fit in with the environment that you need to set up. Mine uses a bash boot script and looks like something like this:
#! /bin/bash
sudo apt-get update
sudo apt-get -yq install python-pip
sudo pip install --upgrade google-cloud
sudo pip install --upgrade google-cloud-storage
sudo pip install --upgrade google-api-python-client
sudo pip install --upgrade google-auth-httplib2
echo "Making directories..."
mkdir -p /home/<username>/rawdata
mkdir -p /home/<username>/processeddata
mkdir -p /home/<username>/input
mkdir -p /home/<username>/code
mkdir -p /home/<username>/and so on
echo "Setting ownership..."
sudo chown -R <username> /home/<username>
echo "Directory creation complete..."
gsutil cp gs://<bucket with code>/* /home/<username>/code/
echo "Boot complete."
nohup python /home/<username>/code/workermaster.py &
The gist of the code (in case it isn't self-explanatory!) is that the instance installs various packages to support the code, but some of these might be on GCE instances by default. It then sets up required directories and copies all required code from a storage bucket and sets ownership. The key line I guess is the one containing "ho hangup" that starts the python script.
My "workermaster" script gets tasks from a Storage bucket and puts the output in one of the folders on the instance. I can see what the instance is doing from the console by looking at the logs (so without SSH-ing into the instance). My instances also copy the output to an output storage bucket. If I SSH into the instance I cannot see the script running, I can just see files 'mysteriously' appearing in the output folder.
There are plenty of experts on here that might be able to post a solution that more specific to your needs, but something like the above has worked for me so far. Good luck!
Not sure why you are saying you have to keep the terminal connected to your compute instance. Some more details will be helpful. Are you manually SSHing into your instance thru terminal and running the script? Is that how you want to do it in future?
If you are running your script periodically, you can set it up as cron job.
You may also want to look at Cloud Functions to go serverless.
You can use programs like tmux.
# ssh to the system
ssh user#system-blah-blah
# start a new tmux session
tmux new -s my_remote_session
# detach from session (keyboard shortcut, in sequence)
<ctrl-b> d
# attach to it back
tmux a -t my_remote_session
To be able to let a script running and close the terminal window, you can use a screen session, this is going to let the script running (inside the screen session) and if you close the terminal, it is going to continue working, after that, you can open again the terminal and connect to the screen session to see the results.
Other option is to use ansible, it helps to run commands inside the VM without connecting to it, but you must create a SSH-key in order to be able to connect with ansible.
Question: How can I confirm whether or not my "Dedicated Server" is running properly?
Background: I am working to get a 'Dedicated CoreNLP Server' running on a stand-alone Linux system. This system is a laptop running CentOS 7. This OS was chosen because the directions for a Dedicated CoreNLP Server specifically state that they apply to CentOS.
I have followed the directions for the Dedicated CoreNLP Server step-by-step (outlined below):
Downloaded CoreNLP 3.7.0 from the Stanford CoreNLP website (not GitHub) and placed/extracted it into the /opt/corenlp folder.
Installed authbind and created a user called 'nlp' with super user privileges and bind it to port 80
sudo mkdir -p /etc/authbind/byport/
sudo touch /etc/authbind/byport/80
sudo chown nlp:nlp /etc/authbind/byport/80
sudo chmod 600 /etc/authbind/byport/80
Copy the startup script from the source jar at path edu/stanford/nlp/pipeline/demo/corenlp to /etc/init.d/corenlp
Give executable permissions to the startup script: sudo chmod a+x /etc/init.d/corenlp
Link the script to /etc/rc.d/: ln -s /etc/init.d/corenlp /etc/rc.d/rc2.d/S75corenlp
Completing these steps is supposed to allow me to run the command sudo service corenlp start in order to run the dedicated server. When I run this command in the terminal I get the output "CoreNLP server started" which IS consistent with the the start up script "corenlp". I then run the start command again and get this same response, which is NOT consistent with the start up script. From what I can tell, if the server is actually running and I try to start it again I should get the message "CoreNLP server is already running!" This leads me to believe that my server is not actually functioning as it is intended to.
Is this command properly starting the server? How can I tell?
Since the "proper" command was not functioning as I thought it should, I used the command sudo systemctl *start* corenlp.service and checked the service's status with sudo systemctl *status* corenlp.service. I am not sure if this is an appropriate way in which to start and stop a 'Dedicated CoreNLP Server' but I can control the service. I just do not know if I am actually starting and stopping my dedicated server.
Can I use systemctl command to operate my Dedicated CoreNLP Server?
Please read the comments below the originally posted question. This was the back and forth between #GaborAngeli and myself which lead my question/problem being solved.
The two critical steps I took in order to get my instantiation of the CoreNLP server running locally on my machine after following all the directions on how to setup a dedicated server, which are outlined on Stanford CoreNLP's webpage, are as follows:
Made two modifications to the "corenlp" start-up script. (1) added sudo to the beginning because the user "nlp" needs permissions to certain files on the system (2) changed the first folder path from /usr/local/bin/authbind to /usr/bin/authbind. authbind installation must've changed since the start up script was written.
nohup su "$SERVER_USER" -c "sudo /usr/bin/authbind --deep java -Djava.net.preferIPv4Stack=true -Djava.io.tmpdir_"$CORENLP_DIR" -cp "$CLASSPATH" -mx15g edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 80"
If you were to attempt to start the server with the change above you would not successfully run server because sudo usage requires a password input. In order to allow sudo privileges without a required password entry you need to edit the sudoers file (I did this under the root user b/c you need permissions to change or even view this document). my sudoers file was located in /etc. There is a part that says ## Allows people in group wheel to run all commands and below that is a section that says ##Same thing without a password. You just need to remove the comment mark (#) form in front of the next line which says %wheel ALL+(ALL) NOPASSWD: ALL. Save this file. BE CAREFUL IN EDITING THIS FILE AS IT MAY CAUSE SERIOUS ISSUES. MAKE ONLY THE NECESSARY CHANGE OUTLINED ABOVE
Those two steps allowed me to successfully run my dedicated server. My system runs on CentOS 7.
HELPFUL TIP: From my discussion with #GaborAngeli I learned that within the 'corenlp' folder (/opt/corenlp if you followed the directions correctly) you can open the stderr.log file to help you in trouble shooting your server. This outputs what you would see if you were to run the server in the command window. If there is an error it is output here too, which is extremely helpful.
I am using linux server from cPanel. Now I need to use one of the linux command to run. And the command is 'timeout'. My command is something as below
$timeout 2s ./myexecutable < input > output
This is just running ok in my linux machine. But I want to run it in server(linux). But it is not running out there. I saw error log using 2>$1 . and it says command not found.
So how do I specify the path. That means how can I know the exact path to run it.
I am very new to this type of server stuff, so please don't be rude :D.. whatever you know please help me to get out of it.
thank you
On Mac, you can use gtimeout from the coreutils package. To install it, run
brew install coreutils
If you need the command to be called "timeout" then you can alias it.
alias timeout="gtimeout"