I have an app that has been successfully running on EC2 for a few years. The system is Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1032-aws x86_64).
It's a small and simple app with low traffic. I had never made any changes to the server itself until today. I wanted to deal with the X packages can be updated. message, so I ran:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
Then I ran sudo reboot. Once rebooted, the app runs perfectly. I can access it as normal via the public URL and look at things, including db (postgresql directly on the server) data with no issues or surprises.
But, when I tried to ssh into the machine again, I couldn't. I do ssh -i "key.pem" -vvv ubuntu#<IP> and get:
debug1: Connecting to <IP> [<IP>] port 22.
debug1: connect to address <IP> port 22: Operation timed out
ssh: connect to host <IP> port 22: Operation timed out
No changes were made to security groups. Also, it's such a small project, that I never setup EC2 Instance Connect or anything like that.
I had the thought of launching a new EC2 and just switching the EBS volumes, thinking EBS would bring the app and data, while the instance itself would have configs and permissions.
I do not understand much about this (clearly), and was surprised to learn that the EBS volume itself seems to be the problem and hold all the cards.
I can switch EBS volumes back and forth between the two EC2 instances. At any given time, whichever one has the newest (and therefore blank) EBS volume attached at /dev/sda1 allows SSH but surely does not run the app. And, vice-versa: Whichever EC2 instance has the original EBS volume runs the app perfectly but keeps me locked out of ssh.
In this scenario, the question is: How can I make one of the EC2 instances bypass this EBS issue and make its own decision about allowing me to connect with ssh?
Or: What is the obvious and/or silly thing I'm missing here?
PS: I do have elastic IP going for all of this, so it doesn't seem like DNS would be the source of the problem.
With John Rotenstein's help, I was able to resolve this.
Here are the core steps:
Phase 1 - Attach and mount additional volume
Per John's comment, it's possible to boot the instance from the "good" volume and then attach and mount the "bad" volume after. This allowed me to explore files and look for issues.
AWS panel
Attach volume to EC2 instance as root by using /dev/sda1 for name
Start the EC2 instance
Attach the other volume after instance has booted
Terminal
SSH into the server
See root volume information:
~$ df -hT /dev/xvda1
Check for mounted volumes:
~$ lsblk
See additional volume information:
~$ df -hT /dev/xvdf1
Switch to root user:
~$ sudo su -
Make a directory to be the mount path:
~$ mkdir /addvol
Mount the additional volume to the path:
~$ mount /dev/xvdf1 /addvol
Check additional volume contents:
~$ ls -la /addvol/home/ubuntu
Now I could see and navigate the additional volume's contents, finding config files, looking at authorized_keys, file permissions, etc.
This article from AWS helped a lot to get me here.
After finally getting to this point, I could not find any problems with the keys, or permissions, etc. John pointed me to this article dealing with Ubuntu's firewall things.
Phase 2 - Dealing with the firewall
I ran some commands from the article and tried to understand how they worked.
Once I grasped it a little, I decided to use an existing reboot script I have on the volume to ensure the firewall was ok with SSH connections.
I updated my existing custom reboot script, adding the following lines:
sudo ufw allow ssh
sudo ufw allow 22
sudo ufw disable
sudo ufw --force enable
Basically it calls to allow for ssh twice, once by name and then by port. I'm a newbie on this stuff and just went for the overkill.
Then it disables and enables the firewall to ensure it runs with these news things configured.
Because sudo ufw enable requires an interaction, I chose to use sudo ufw --force enable.
Phase 3 - Testing and using it!
After the script update, I exited the server.
AWS panel:
Stop the EC2 instance
Detach one volume from the instance
Detach the other volume from the instance
Reattach the "bad" volume, this time as root
Start the EC2 instance
Terminal:
SSH into the instance - Voila!
NOTE: Before truly working 100%, my computer complained about the known_hosts thing. The server key must have changed on the update/upgrade and/or after all of the volume changes. I don't think having to confirm hosts is a big deal, so I just usually clear all of the contents in my local .ssh/known_hosts file. If you prefer to be specific, you can find the server's information on there specifically and delete only the relevant lines.
I have installed cassandra 3.11.4 as per the below link http://cassandra.apache.org/download/ and when I'm trying to run cqlsh and nodetool command its showing command not found.
when run sudo service cassandra status,cassandra is running.
Please help
First, you need to verify that the service is healthy and operational
verify that the process is running:
ps -efa | grep cassandra
The expected outcome is a valid java process being executed
verify the log file as defined in $CASSANDRA_HOME/conf/logback.xml
tail -n 100 cassandra/system.log
The expected outcome is to not find errors
Try to replicate the issue in a new instance of the command line
You need to verify the services whether it is running or not in the configured port and IPs.
Also, you can check the system.log for any error. if still issue and data is not important you can delete the /data /commitlog /savedcache and restart the cassandra services again.
once started just check
ps -ef | grep cassandra
I installed Cassandra on OSX.
When trying to run cqlsh from /bin/, I get the following error:
> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
> ConnectionShutdown('Connection <AsyncoreConnection(4522252560)
> 127.0.0.1:9160 (closed)> is already closed',)})
I had difficulty installing Cassandra, and had used brew install cassandra, installed DSE from datastax, and downloaded a binary from Apache.
I think the issue was the result of having a previous installation that wasn't properly uninstalled. My user .cassandra directory had some default configs, that must have been for a different version, and prevented cqlsh from starting.
rm -Rf ~/.cassandra
After that, I could run cqlsh properly.
$ bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh>
Running cassandra helped me to troubleshot the problem:
> cassandra
Cassandra 3.0 and later require Java 8u40 or later.
and:
> java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
After installing Java 8 it worked.
Cassandra-2.1 uses 9042 as cqlsh port.Connect on this it is default port.If you want to connect on 9160 port then change it by setting environment variable CQLSH_PORT and CQLSH_HOST but for setting this you have to change rpc_port from default 9160 to any other port.
Well, I faced same issue and this is what i did on Mac.
$cd /usr/local/Cellar/cassandra/3.10_1/bin/
Or wherever your cassandra is installed.
$cassandra
this initializes all configuration in your local system to start cassandra.
$./cqlsh
This lets you enter cql query mode.
I resolved this in DSE-5.0 by editing below file:
vi ~/.cassandra/cqlshrc
And editing below configs to current IP address (192.168.1.6) of the machine I am running. In my case it's my MacBook.
[connection]
hostname = 192.168.1.6
port = 9042
; factory = cqlshlib.kerberos_ssl.kerberos_ssl_transport_factory
; [kerberos]
; hostname = 192.168.1.6
; service = cassandra
; optional
; principal = bill/cassandra-admin#foo.example.com
I hope it will help someone.
Thanks,
Chandan
I faced this issue when I was trying to connect to Cassandra 2.0.1 version using cqlsh 5.0.1
[cqlsh 4.1.1 | Cassandra 2.0.11.83 | CQL spec 3.1.1 | ]
[cqlsh 5.0.1 | Cassandra 2.1.2 | CQL spec 3.2.0 ]
This is just one of the reasons for the error "Connection error: ('Unable to connect to any servers',".
When I used the cqlsh client compatible with Cassandra 2.0.1, I was able to connect to the cluster.
Hope this helps to find the root cause.
I got the same error after moving from Java 8 to Java 11. From https://github.com/nosan/embedded-cassandra/issues/21, Apache Cassandra does not yet have support for java 11.
Reverting to the system's current java version to Java 8 solved the problem.
Just add/modify the following into the cassandra.yaml:
rpc_address: 127.0.0.1
seeds: "127.0.0.1"
listen_address: 127.0.0.1
Note that, the values in all these fields must be same.
That means, if you are pointing to a static IP, then the same IP should be used. And if you are just pointing the localhost, then localhost must be used in all fields.
I got myself into a mess so here was my scenario if anyone else runs into it:
I followed this tutorial
http://www.datastax.com/2012/01/working-with-apache-cassandra-on-mac-os-x
(which is somewhat outdated).
I got everything installed and working - all good. Then I decided to download Datastax Enterprise tools to try them out
Ran into errors during the install but the install went through. Again following the tutorial above, tried running the ./datastax-agent program and it failed. Decided to just bag it so ran their uninstaller in their installation folder.
Tried to go back to original cassandra folder (what I had working in step 2) and run "sudo ./cassandra" then I started getting JMX error (message "Unable to bind JMX, is Cassandra already running?") which led me to this link about changing the port which i did (not really fixing the issue but working around it) run two cassandra versions in the same machine.
Got the server service running underneath a new port but when trying to connect with the client "sudo ./cqlsh" would not work (would get "Error=none" or destination not found).
Did the remove step #Peter mentioned above and tried again and still did not work.
Decided to go back and see what was holding onto the original port cassandra was using which I changed in step 4 Who is listening on a given TCP port on Mac OS X?
Determined that JAVA.app was hanging onto it so killed it within Activity Monitor.
Reran the remove command #Peter mentioned above again and decided to download the stable release (v2.0.13) instead of the dev release and walk through the tutorial from step 1 again.
Exited and re-opened terminal. Ran "sudo ./cassandra" - worked perfect running under standard ports. Open another terminal window and ran "sudo ./cqlsh" and this also worked correctly.
I believe between Java hanging onto the default port 9042 (maybe from the DataStax install errors), version/host/port settings within ~/.cassandra/cqlshrc and using the dev version of cassandra these were all causing me fits.
I was having trouble connecting Cassandra from the terminal
[
and then I tried to connect using /usr/local/Cellar/
[
and then invoke cassandra. Then opened a new terminal and typed cqlsh the Cassandra started.
Hope this may also applicable one to resolve Cassandra issue.
install java jdk and Set JAVA_HOME in Environment Variables or set JAVA_HOME = java path in command prompt.
after
run cassandra.bat
it will show the logs if you get any error
You need to start Cassandra form the install_location's bin directory.
For example, this command will start Cassandra in the foreground:
$ ./cassandra -f
Then open another terminal tab (command + t) and run cqlsh
$ ./cqlsh
Ensure that DEFAULT_HOST = '127.0.0.1' in script cqlsh.py is pointing to the node IP where you have done your cassandra installation.
In my case I had installed cassandra on 10.160.1.5, edited above file and it started working.
Running cqlsh gets this error for me but running bin/cqlsh from the cassandra directory works.
I faced the same issue. I ran cassandra.bat --> This appears to have started the server. Then I opened a new command prompt (run as an administrator), changed directory to the cassandra/bin location. When I ran the command cqlsh, voila it worked!
I solve this issue by just simply restarting cassandra sudo systemctl restart cassandra
In case of MacOSX, it worked to just run
cassandra
rather than running it through brew services start cassandra.
I was having the same problem with Cassandra configured with SSL.
cqlsh --ssl
Solves connects correctly.
I am trying to setup cassandra cluster with 5 nodes. I have installed dse on all nodes and started dse on all the nodes by command.
sudo service dse start
dse is running fine on all nodes.
Now I am trying to configure opscenter following http://www.datastax.com/documentation/opscenter/3.2/webhelp/index.html#opsc/install/../../opsc/install/opscInstallRHEL_t.html
When I execute "sudo service opscenterd start", it starts without any problem and even log doesnt show any problem.
But when I try doing "netstat -a | grep 8888", it doesnt show any listener.
Can anybody please help me in identifying issue?
Thanks,
Jenish
I would first figure out if the service is indeed starting. When you say you checked log, was that /var/log/messages or the opscenter logs? I would check both.
Next I would see if it stays running. You can also check for the process running with
ps -eaf | grep opscenterd
If everything is running but not listening on the right port, you should check your opscenterd.conf file for proper port and interface:
[webserver]
port = 8888
interface = 127.0.0.1
Note that your interface definition may be different - for example, it may be 0.0.0.0 which signifies binding to all interfaces (rather than just localhost as above), but you should validate that it is correct for your environment.
I'm trying to install and configure cassandra programmatically using a shell script.
I install cassandra, run ./cassandra and then I try to load the schema using cassandra-cli --host localhost -f <schema-file>. The problem is that it tries to load the schema before cassandra is up and throws an exception. Is there any way to know when cassandra is up and running?
Thanks!
You can check to see if anything is listening on 9160 (the Thrift port) prior to loading the schema. Binding to the port is the last thing the service does, and indicates it's ready to service requests.
If Cassandra is running this command will return something:
sudo lsof -i :9160
use ./cassandra -f then you will know it when Cassandra starts to listen on the Thrift port