Fastest way to stop and start cassandra nodes before updates - cassandra

What would be the best way how to stop Cassandra nodes before application updates and database updates, and to afterwards start them? Currently I'm going through each node VM and stopping them using the command sudo service cassandra stop and then after the update is done, I'm doing the same, just starting them with command sudo service cassandra start. With few nodes it's OK, but I have 20 nodes on a server, and it takes some time to do that.
Is there any better/more efficient way to do that?

There are so many ways to do this - use for loop in shell + ssh, use pssh, etc. You just need to remember that it's better to perform rolling restart of the individual Cassandra nodes, so you need to have to wait until Cassandra starts again. Something like:
for host in `cat your-host-list.txt`; do
ssh user#${host} 'sudo service cassandra stop && your_update_command && sudo service cassandra start
done
or slightly easier with pssh:
pssh -l user -h your-host-list.txt -p 1 -t 0 'sudo service cassandra stop && your_update_command && sudo service cassandra start
-p 1 tells to execute only one command in parallel
t 0 instructs to wait for execution without timeout

Related

How to list/kill running commands in cassandra

I want to run a COPY command on my Cassandra cluster (v3.0.9).
In case my shell exits, how can I later on list running commands, get their status and possibly kill/stop it?
Create a cql file with your COPY command
Run it through cqlsh using -f flag (cqlsh -f ) with nohup. Details here.
Then it will be available to ps and kill.

Cassandra Apache : Problems execution cassandra -f

I would to know if it's an obligation to execute cassandra -f for use cassandra-cli or cqlsh ??? Because, i tested connection to cassandra-cli --host 127.0.0.1 --port 9160 or cqlsh and i have a message error : connection failed to localhost/9160
The -f option is a FLAG, so no you don't have to supply it when starting cassandra.
What does -f get used for?
Depends on the OS but its used for choosing whether to run the process in the foreground / background.
In Windows
-f means you will run the process in the background. Simply calling cassandra.bat means the process starts in the foreground. E.g:
cassandra.bat # cassandra runs in foreground (i.e.) you see logdata in the terminal
cassandra.bat -f # cassandra runs in background
In UNIX based OSs (linux / osx)
Supplying the -f flag means you run the process in the foreground. Not supplying it means the process will start in the background. E.g:
cassandra.bat # cassandra runs in background, you wont see anything in the terminal
cassandra.bat -f # cassandra runs in foreground
Back to the error
This means that cassandra isn't running. Start the task manager / activity monitor and verify that Cassandra is executing as a process / service. And also if you want to connect cqlsh / cassandra-cli to a local server without specifying the localhost ip address you can simply type:
path/to/cassandra/bin/cqlsh -3 # the -3 means you use CQL3
path/to/cassandra/bin/cassandra-cli

Running Housekeeper scripts on a HA (Active/Passive)

I have following scenario. previously I had only one server running the application and it had housekeeper script running to archive log files. Now we have implemented HA ( high availability) and, I want to know in this case, how the script should identify which server is running currently and then to run the script accordingly, HA is implemented in active/passive mode. There are 2 physical servers to support HA, if one is down , other one will be active.
Use this Shell ecript for high availability on passive server.
#!/bin/bash
crumb_id=$(curl -s 'http://locathost:8080/crumbIssuer/api/xml?xpath=contact(//crumbRequestField,":",//crumb)' -u admin:admin)
curl -s -XPOST 'http://locathost:8080/reload' -u admin:admin -H "$crumb_id"
~

stop cassandra server on mac os x

How do I stop cassandra server running on a single node in my mac os x?
Cassandra script doesn't have -stop option. Only way other than restart the mac os x, was to do a "ps" and find the java process which had arguments for cassandra and use kill -9 to kill the process.
But trying to restart cassandra after that still throws
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use.
Anybody seen it? Any quick solutions?
If you've installed cassandra via homebrew, use brew info cassandra and it will tell you how to load/unload cassandra using launchctl. This worked better for me than the other answers here.
Commands
brew info cassandra To see status of cassandra
brew services start cassandra To start cassandra
brew services stop cassandra To stop cassandra
EDIT: I actually find this much more useful.
Open terminal and type:
$ ps -ax | grep cassandra
gives you a list of pids running with the name cassandra.
Use the PID number to kill the process for example here is a returned value:
708 ttys000 0:03.10 /usr/bin/java -ea -javaagent:Downloads/Web/Cassandra/dsc-cassandra-1.1.0/bin/
$ kill 708
Old post:
After posting my comment I found a stop-server script in the BIN.
You have to open up the script and comment out the code if you want to use that script. But here is what it says inside the script.
echo "please read the stop-server script before use"
# if you are using the cassandra start script with -p, this
# is the best way to stop:
kill `cat <pidfile>`
# otherwise, you can run something like this, but
# this is a shotgun approach and will kill other processes
# with cassandra in their name or arguments too:
# user=`whoami`
# pgrep -u $user -f cassandra | xargs kill -9
Found this solution elsewhere which seems to work!
pkill -f 'java.*cassandra'
Worth a try!
This works on the Ubuntu I have. Not on MacOS!
On Mac one more is ps -af | grep cassandra and then using kill. But, it does not work sometimes!
Another approach is to see which OS process has the Cassandra port open, like this:
lsof -i :9160
Sample output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 30253 aswan 214u IPv4 0xffffff80190dcc20 0t0 TCP *:netlock1 (LISTEN)
Then you can use "kill -9 [pid]" on that process.
You can use Cassandra's nodetool command, as well.
nodetool drain
The documentation doesn't say anything about it shutting down, but it works reliably for me with a single node, local server. It generally takes a few seconds to finish the shutdown, however.
kill -9 ` acx | grep -i cassandra | awk '{print$1}' `
I'm using the new Datastax Enterprise 5.0 version, and it, at least, offers a simple enough command for stopping Cassandra:
dse cassandra-stop
It takes a few moments to shut down, but it works for me.

How to make sure an application keeps running on Linux

I'm trying to ensure a script remains running on a development server. It collates stats and provides a web service so it's supposed to persist, yet a few times a day, it dies off for unknown reasons. When we notice we just launch it again, but it's a pain in the rear and some users don't have permission (or the knowhow) to launch it up.
The programmer in me wants to spend a few hours getting to the bottom of the problem but the busy person in me thinks there must be an easy way to detect if an app is not running, and launch it again.
I know I could cron-script ps through grep:
ps -A | grep appname
But again, that's another hour of my life wasted on doing something that must already exist... Is there not a pre-made app that I can pass an executable (optionally with arguments) and that will keep a process running indefinitely?
In case it makes any difference, it's Ubuntu.
I have used a simple script with cron to make sure that the program is running. If it is not, then it will start it up. This may not be the perfect solution you are looking for, but it is simple and works rather well.
#!/bin/bash
#make-run.sh
#make sure a process is always running.
export DISPLAY=:0 #needed if you are running a simple gui app.
process=YourProcessName
makerun="/usr/bin/program"
if ps ax | grep -v grep | grep $process > /dev/null
then
exit
else
$makerun &
fi
exit
Then add a cron job every minute, or every 5 minutes.
Monit is perfect for this :)
You can write simple config files which tell monit to watch e.g. a TCP port, a PID file etc
monit will run a command you specify when the process it is monitoring is unavailable/using too much memory/is pegging the CPU for too long/etc. It will also pop out an email alert telling you what happened and whether it could do anything about it.
We use it to keep a load of our websites running while giving us early warning when something's going wrong.
-- Your faithful employee, Monit
Notice: Upstart is in maintenance mode and was abandoned by Ubuntu which uses systemd. One should check the systemd' manual for details how to write service definition.
Since you're using Ubuntu, you may be interested in Upstart, which has replaced the traditional sysV init. One key feature is that it can restart a service if it dies unexpectedly. Fedora has moved to upstart, and Debian is in experimental, so it may be worth looking into.
This may be overkill for this situation though, as a cron script will take 2 minutes to implement.
#!/bin/bash
if [[ ! `pidof -s yourapp` ]]; then
invoke-rc.d yourapp start
fi
If you are using a systemd-based distro such as Fedora and recent Ubuntu releases, you can use systemd's "Restart" capability for services. It can be setup as a system service or as a user service if it needs to be managed by, and run as, a particular user, which is more likely the case in OP's particular situation.
The Restart option takes one of no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always.
To run it as a user, simply place a file like the following into ~/.config/systemd/user/something.service:
[Unit]
Description=Something
[Service]
ExecStart=/path/to/something
Restart=on-failure
[Install]
WantedBy=graphical.target
then:
systemctl --user daemon-reload
systemctl --user [status|start|stop|restart] something
No root privilege / modification of system files needed, no cron jobs needed, nothing to install, flexible as hell (see all the related service options in the documentation).
See also https://wiki.archlinux.org/index.php/Systemd/User for more information about using the per-user systemd instance.
I have used from cron "killall -0 programname || /etc/init.d/programname start". kill will error if the process doesn't exist. If it does exist, it'll deliver a null signal to the process (which the kernel will ignore and not bother passing on.)
This idiom is simple to remember (IMHO). Generally I use this while I'm still trying to discover why the service itself is failing. IMHO a program shouldn't just disappear unexpectedly :)
Put your run in a loop- so when it exits, it runs again... while(true){ run my app.. }
I couldn't get Chris Wendt solution to work for some reason, and it was hard to debug. This one is pretty much the same but easier to debug, excludes bash from the pattern matching. To debug just run: bash ./root/makerun-mysql.sh. In the following example with mysql-server just replace the value of the variables for process and makerun for your process.
Create a BASH-script like this (nano /root/makerun-mysql.sh):
#!/bin/bash
process="mysql"
makerun="/etc/init.d/mysql restart"
if ps ax | grep -v grep | grep -v bash | grep --quiet $process
then
printf "Process '%s' is running.\n" "$process"
exit
else
printf "Starting process '%s' with command '%s'.\n" "$process" "$makerun"
$makerun
fi
exit
Make sure it's executable by adding proper file permissions (i.e. chmod 700 /root/makerun-mysql.sh)
Then add this to your crontab (crontab -e):
# Keep processes running every 5 minutes
*/5 * * * * bash /root/makerun-mysql.sh
The supervise tool from daemontools would be my preference - but then everything Dan J Bernstein writes is my preference :)
http://cr.yp.to/daemontools/supervise.html
You have to create a particular directory structure for your application startup script, but it's very simple to use.
first of all, how do you start this app? Does it fork itself to the background? Is it started with nohup .. & etc? If it's the latter, check why it died in nohup.out, if it's the first, build logging.
As for your main question: you could cron it, or run another process on the background (not the best choice) and use pidof in a bashscript, easy enough:
if [ `pidof -s app` -eq 0 ]; then
nohup app &
fi
You could make it a service launched from inittab (although some Linuxes have moved on to something newer in /etc/event.d). These built in systems make sure your service keeps running without writing your own scripts or installing something new.
It's a job for a DMD (daemon monitoring daemon). there are a few around; but I usually just write a script that checks if the daemon is running, and run if not, and put it in cron to run every minute.
Check out 'nanny' referenced in Chapter 9 (p197 or thereabouts) of "Unix Hater's Handbook" (one of several sources for the book in PDF).
A nice, simple way to do this is as follows:
Write your server to die if it can't listen on the port it expects
Set a cronjob to try to launch your server every minute
If it isn't running it'll start, and if it is running it won't. In any case, your server will always be up.
I think a better solution is if you test the function, too. For example, if you had to test an apache, it is not enough only to test, if "apache" processes on the systems exist.
If you want to test if apache OK is, then try to download a simple web page, and test if your unique code is in the output.
If not, kill the apache with -9 and then do a restart. And send a mail to the root (which is a forwarded mail address to the roots of the company/server/project).
It's even simplier:
#!/bin/bash
export DISPLAY=:0
process=processname
makerun="/usr/bin/processname"
if ! pgrep $process > /dev/null
then
$makerun &
fi
You have to remember though to make sure processname is unique.
One can install minutely monitoring cronjob like this:
crontab -l > crontab;echo -e '* * * * * export DISPLAY=":0.0" && for
app in "eiskaltdcpp-qt" "transmission-gtk" "nicotine";do ps aux|grep
-v grep|grep "$app";done||"$app" &' >> crontab;crontab crontab
disadvantage is that the app names you enter have to be found in ps aux|grep "appname" output and at same time being able to be launched using that name: "appname" &
also you can use the pm2 library.
sudo apt-get pm2
And if its a node app can install.
Sudo npm install pm2 -g
them can run the service.
linux service:
sudo pm2 start [service_name]
npm service app:
pm2 start index.js

Resources