I want to run a COPY command on my Cassandra cluster (v3.0.9).
In case my shell exits, how can I later on list running commands, get their status and possibly kill/stop it?
Create a cql file with your COPY command
Run it through cqlsh using -f flag (cqlsh -f ) with nohup. Details here.
Then it will be available to ps and kill.
Related
I just started using slurm and did
salloc -p UBUNTU bash
I started a script, then my system froze for another reason and I had to restart. How can I retrieve the ID of the allocated job so I can reattach and end the script?
You can see the list of your jobs with:
squeue -u $USER
Assuming you connect to your cluster with SSH, if you did not start a terminal multiplexer (such as screen or tmux) your job was most probably killed as soon as your SSH session ended.
I have a scala script file that gets successfully executed via interactive spark-shell in a classic way: type spark-shell, paste script, wait till completion.
I want to be capable to leave this thing working and exit ssh session, get back to results when I need.
I tried this and it behaves strangely
spark-shell -i file.scala >> out.log 2>&1 &
It prints only several lines of usual spark output to out.log and then reports that the process has ended. When I do 'ps aux | grep spark' I see there is spark running among processes.
When I run this it behaves as expected, but I have to leave session open to have my results.
spark-shell -i file.scala
Is there a way to get spark-shell working with nohup properly?
I know there is spark-submit working with jars but it feels less intuitive, for a simple test I have to asseble a jar and do maven magic.
I encountered the same behavior of spark-shell with nohup. The reasons behind this are unclear, but one can use tmux instead of nohup as a work-around. A pretty good guide on how to use tmux can be found here.
Possible set of actions is as following:
$ tmux new -s session-name
$ ./bin/spark-shell
# do usual stuff manually
Then if you close the terminal window and exit ssh session, you can re-enter the tmux session like this:
$ tmux attach -t session-name
I use a shell script to execute spark-shell, inside my-script.sh:
$SPARK_HOME/bin/spark-shell < $HOME/test.scala > $HOME/test.log 2>&1 &
Read it somewhere by googling and tried it. It is working on my end.
If you are trying to execute it on aws-cli then you use the below command ..
nohup bash -c "YOUR_COMMAND 2>&1 &"
So to execute the spark-shell
nohup bash -c "spark-shell -i file.scala >> out.log 2>&1 &"
Old question, but did you actually try to use the nohup command?
Simply using & to background a process does not prevent it from exiting if it receives a SIGHUP signal, which is what the process will receive when you log out.
Try this:
nohup spark-shell -i file.scala >> out.log &
I'm a bit late to the party, but I recently discovered another solution:
echo ":load myscript.scala" | nohup $SPARK_HOME/bin/spark-shell [other args]
where other args represent more arguments passed to the spark-shell (not to your script; I haven't tested that part). I have a df.write() call at the end of the script so the results are saved to HDFS - no need to have them printed on screen. Note I don't need an & at the end of the command.
I have tried closing the SSH connection and the spark-shell job keeps running tasks, according to the Spark UI :-)
I am trying to execute the CQL commands from shell script.
I am able to connect to the cqlsh (CQL version i'm using is 1.1.18) but unable to send the queries to cql.
Any ideas or suggestion how to proceed on this?
Do I need to connect to Cassandra and execute few commands (select/update ) with shell script ??
cqlsh -e "select * from ks.table limit 1;" > ~/output
I'm not sure about Cassandra 1.1.18, but you should be able to accomplish this with the -f flag of cqlsh. Let's say have a file of CQL commands called "commands.cql". I can invoke those commands against my local Cassandra instance like this:
$ cqlsh -f commands.cql -u myusername -p mypassword localhost
If I wanted to invoke that from within a Bash script, the script's code would look something like this:
#!/bin/bash
cqlsh -f commands.cql -u myusername -p mypassword localhost
Save that as an executable file, and run it like any other.
Need to connect to cassandra and execute few commands (select / update ) with shell script
You can execute your commands with shell script in next way:
echo "some QUERY; exit" | cqlsh CASSANDRA_HOST -u 'USER' -p 'PASS'
The "exit" command in the last suggestion is a bit hacky.
I would propose using xargs with cqlsh -e.
echo "some QUERY;" | xargs cqlsh CASSANDRA_HOST -u 'USER' -p 'PASS' -e
I recently had to use this approach when working with docker, because clqsh -f was not an option (too complex to configure access to the file needed).
echo "some QUERY;" | xargs cqlsh CASSANDRA_HOST -u 'USER' -p 'PASS' -e
But what if you Cassandra instance is on a different server to where the shell script is being executed? (Specifically in StreamSets - wouldn't the above require Cassandra installed on the same server such that it has access to the cqlsh lib?)
I would to know if it's an obligation to execute cassandra -f for use cassandra-cli or cqlsh ??? Because, i tested connection to cassandra-cli --host 127.0.0.1 --port 9160 or cqlsh and i have a message error : connection failed to localhost/9160
The -f option is a FLAG, so no you don't have to supply it when starting cassandra.
What does -f get used for?
Depends on the OS but its used for choosing whether to run the process in the foreground / background.
In Windows
-f means you will run the process in the background. Simply calling cassandra.bat means the process starts in the foreground. E.g:
cassandra.bat # cassandra runs in foreground (i.e.) you see logdata in the terminal
cassandra.bat -f # cassandra runs in background
In UNIX based OSs (linux / osx)
Supplying the -f flag means you run the process in the foreground. Not supplying it means the process will start in the background. E.g:
cassandra.bat # cassandra runs in background, you wont see anything in the terminal
cassandra.bat -f # cassandra runs in foreground
Back to the error
This means that cassandra isn't running. Start the task manager / activity monitor and verify that Cassandra is executing as a process / service. And also if you want to connect cqlsh / cassandra-cli to a local server without specifying the localhost ip address you can simply type:
path/to/cassandra/bin/cqlsh -3 # the -3 means you use CQL3
path/to/cassandra/bin/cassandra-cli
How do I stop cassandra server running on a single node in my mac os x?
Cassandra script doesn't have -stop option. Only way other than restart the mac os x, was to do a "ps" and find the java process which had arguments for cassandra and use kill -9 to kill the process.
But trying to restart cassandra after that still throws
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use.
Anybody seen it? Any quick solutions?
If you've installed cassandra via homebrew, use brew info cassandra and it will tell you how to load/unload cassandra using launchctl. This worked better for me than the other answers here.
Commands
brew info cassandra To see status of cassandra
brew services start cassandra To start cassandra
brew services stop cassandra To stop cassandra
EDIT: I actually find this much more useful.
Open terminal and type:
$ ps -ax | grep cassandra
gives you a list of pids running with the name cassandra.
Use the PID number to kill the process for example here is a returned value:
708 ttys000 0:03.10 /usr/bin/java -ea -javaagent:Downloads/Web/Cassandra/dsc-cassandra-1.1.0/bin/
$ kill 708
Old post:
After posting my comment I found a stop-server script in the BIN.
You have to open up the script and comment out the code if you want to use that script. But here is what it says inside the script.
echo "please read the stop-server script before use"
# if you are using the cassandra start script with -p, this
# is the best way to stop:
kill `cat <pidfile>`
# otherwise, you can run something like this, but
# this is a shotgun approach and will kill other processes
# with cassandra in their name or arguments too:
# user=`whoami`
# pgrep -u $user -f cassandra | xargs kill -9
Found this solution elsewhere which seems to work!
pkill -f 'java.*cassandra'
Worth a try!
This works on the Ubuntu I have. Not on MacOS!
On Mac one more is ps -af | grep cassandra and then using kill. But, it does not work sometimes!
Another approach is to see which OS process has the Cassandra port open, like this:
lsof -i :9160
Sample output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 30253 aswan 214u IPv4 0xffffff80190dcc20 0t0 TCP *:netlock1 (LISTEN)
Then you can use "kill -9 [pid]" on that process.
You can use Cassandra's nodetool command, as well.
nodetool drain
The documentation doesn't say anything about it shutting down, but it works reliably for me with a single node, local server. It generally takes a few seconds to finish the shutdown, however.
kill -9 ` acx | grep -i cassandra | awk '{print$1}' `
I'm using the new Datastax Enterprise 5.0 version, and it, at least, offers a simple enough command for stopping Cassandra:
dse cassandra-stop
It takes a few moments to shut down, but it works for me.