I am trying to install and run the Datastax cassandra community edition on Redhat Linux but I don't have root privileges. I extracted the tar in my home directory but I'm unable to do ./cassandra
I am doing this on a HPC cluster and thought I'd install Cassandra in my home directory and save the data in a scratch space we've been provided (home directory doesn't have enough space to hold entire data)
I would appreciate any help! Thanks!
From the installation docs for DataStax community edition, the only other step you need is to create the data and log directories:
$ sudo mkdir /var/lib/cassandra
$ sudo mkdir /var/log/cassandra
$ sudo chown -R $USER: $GROUP /var/lib/cassandra
$ sudo chown -R $USER: $GROUP /var/log/cassandra
If you are using a different location, that's fine. Just make sure to create the dirs and assign owners (like above) and also set the appropriate values in cassandra.yaml (data_file_directories, commitlog_directories, saved_caches_directory) and log4j-server.properties.
A more detailed log of the results you're seeing would confirm whether this is the problem.
Yes, you can run Cassandra without having root or sudo privileges. Extract Cassandra tar file into your local user directory, configure cassandra.yaml as single node. Then you run Cassandra from bin directory, either in foreground or background and login using cql shell.
bin/cassandra -f
OR
bin/cassandra
AND
cqlsh
This is for Cassandra version 2.1x
You can run Cassandra without root or sudo privileges, Besides extracting the
tar file, you need to modify the conf/logback.xml to redirect the log to
your home or somewhere you can write.
<file>/home/xxxx/system.log</file>
<fileNamePattern>/home/xxxx/system.log.%i.zip</fileNamePattern>
The only minor issue of not running with root is - the ULIMIT -l (RLIMIT on
max locked memory) will need to be increased and I cannot increase it with my account.
But this does not prevent it to run..
In my opinion, almost all the java-written apache projects need not the root privilege. Cassandra has the same feature.
Firstly, download apache-cassandra-bin.tar.gz from http://cassandra.apache.org/download/. Remember that do not use .deb or .rpm or others.
Secondly, run tar -xzf cassandra-bin.tar.gz to unzip it to any folder, suppose the folder is $cassandra_home
Thirdly, just go to $cassandra_home/bin, run ./cassandra, done! The data is stored in $cassandra_home/data and the logs are in $cassandra_home/logs.
If you want to set the position of data and logs:
1st, go to $cassandra_home/conf, modify cassandra.yaml file.
Set these directories to the folder which you have read and write access:
data_file_directories:
commitlog_directory:
cdc_raw_directory:
hints_directory:
saved_caches_directory:
(different cassandra version may have different parameters. You can just search director in the yaml file.)
2nd, if you want to enable the log, modify the log file position, modify $cassandra_home/conf/logback.xml (or log4j or others), and set the log folder to another position.
Enjoy it.
Related
In order to test and learn Spark functions, developers require Spark latest version. As the API's and methods earlier to version 2.0 are obsolete and no longer work in the newer version. This throws a bigger challenge and developers are forced to install Spark manually which wastes a considerable amount of development time.
How do I use a later version of Spark on the Quickstart VM?
Every one should not waste setup time which I have wasted, so here is the solution.
SPARK 2.2 Installation Setup on Cloudera VM
Step 1: Download a quickstart_vm from the link:
Prefer a vmware platform as it is easy to use, anyways all the options are viable.
Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids.
Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.
Step 3: Please open the terminal and switch to root user as:
su root
password: cloudera
Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
Downloading Java:
wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
Switch to /usr/java/ directory with “cd /usr/java/” command.
cp the java download tar file to the /usr/java/ directory.
Untar the directory with “tar –zxvf jdk-8u31-linux-x64.tar.gz”
Open the profile file with the command “vi ~/.bash_profile”
export JAVA_HOME to the new java directory.
export JAVA_HOME=/usr/java/jdk1.8.0_131
Save and Exit.
In order to reflect the above change, following command needs to be executed on the shell:
source ~/.bash_profile
The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.
Switch to /opt/ directory with the command:
cd /opt/
Download spark with the command:
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
Untar the spark tar with the following command:
tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz
We need to define some environment variables as default settings:
Please open a file with the following command:
vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
Paste the following configurations in the file:
SPARK_MASTER_IP=192.168.50.1
SPARK_EXECUTOR_MEMORY=512m
SPARK_DRIVER_MEMORY=512m
SPARK_WORKER_MEMORY=512m
SPARK_DAEMON_MEMORY=512m
Save and exit
We need to start spark with the following command:
/opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
Export spark_home :
export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/
Change the permissions of the directory:
chmod 777 -R /tmp/hive
Try “spark-shell”, it should work.
Can not login to system after start server:
openpam_check_desc_owner_perms() : /etc/pam.d/login insecure perms
Started system in single-user mod. There are all system dirs (/boot , /lib ...)
nobody:nogroup
ownership.
Ofc, i cannot just chown system dirs. So, how can i restore ownership to root?
Firstly, you should use a live boot downloaded from official repository. I don't think rescue mode (in this case) can save you.
In live the live environment, just mount your impacted filesystems somewhere:
mount /dev/${your_partition} /mnt
Now, you can use mtree to set good rights on all default files and directory. This command will re-create your tree with good rights. Before executing it, you can run it to check state of your filesystem.
# check before act
mtree -f /etc/mtree/BSD.root.dist -p /mnt
# you can now apply change
mtree -u -f /etc/mtree/BSD.root.dist -p /mnt
You can find more mtree files in /etc/mtree:
BSD.debug.dist
BSD.include.dist
BSD.sendmail.dist
BSD.usr.dist
BSD.var.dist
After doing that, you can now run mergemaster. mergemaster will warn when some files aren't well configured with good rights:
mergemaster -iD /mnt
If you have still issue, you can download or fetch FreeBSD source from SVN, extract them, and reinstall your configuration files manually (do a backup of your configuration before doing that and please read official FreeBSD documentation about using source).
cd /usr/src
svnlite https://svnweb.freebsd.org/base/releng/${your_freebsd_release} .
cd /usr/src/etc
make install DESTDIR=/mnt
You can now reboot your computer or server without live CD.
Working with memsql cluster as primary storage design, by default data files are installed in a place like the following on CentOS 6.x:
/var/lib/memsql-ops/data/installs/MI9dfcc72a5b044f2694b5f7028803a21e
Is there any way to relocate the data path to another folder on the same machine?
This is not a best way but it works. I just re-install MemSQL to other directory:
sudo mkdir /data/memsql
sudo ./install.sh --root-dir /data/memsql
In this case MemSQL Ops still will be in /var/lib/memsql-ops but all nodes will be installed to /data/memsql directory (look at symlink /var/lib/memsql) and all data will be inside this directory too.
P.S. Additional installation options you can find use memsql-ops agent-install --help command.
Am currently working on a CouchDB project, and have recently decided to switch to a Linux environment for development as I plan to deploy on a Linux server.
I was hoping to copy over my .couch files straight from - Program Files/Apache Software Foundation/CouchDB-1-1-1/var/lib/couchdb - and paste them directly into what I guess should be - var/lib/couchdb - But I keep running into file/folder permission errors each time I try to access var/lib/couchdb.
Is it even possible to transfer .couch files in the way I envisage?
...
Update - Follwing up on Dominic's comments, I managed to apply the fix found in the answer below.
After some investigative work, I found it to be a permissions error, exactly as Dominic Barnes had suggested in the comments...
The issue is also discussed here - Staging setup with couchdb
To fix it, I first ran;
sudo chmod -R 755 var/lib/couchdb
I may have also changed the permissions on the relevant parent folders too. I was then able to copy my .couch files into var/lib/couchdb/COUCH-VERSION-NUMBER. After doing that, I then had to use chmod to set favourable write permissions on the newly copied files, but also had to run:
sudo chown couchdb var/lib/couchdb/COUCH-VERSION-NUMBER/
To open those files up to the user group (the "couchdb" group) that the couchdb installation sets up for internal use (I think...). After that, I restarted couchdb, forcing it to stop with:
ps -U couchdb -o pid= | xargs kill -9
and restarting with:
/etc/init.d/couchdb start
After that, everything seemed to work as expected.
Hope that helps anyone else running into the same problem.
We're trying to use oprofile to track down performance problems on a server cluster. However, the servers in question have a read-only file system, where /var/tmp is the only writeable directory.
OProfile wants to create two directories whenever it runs: /root/.oprofile and /var/lib/oprofile, but it can't, because the filesystem is read-only. I can use the --session-dir command line option to make it write its logs to elsewhere than /var/lib, but I can't find any such option to make it use some other directory than /root/.oprofile.
The filesystem is read-only because it is on nonwriteable media, not because of permissions -- ie, not even superuser can write to those directories. We can cook a new ROM image of the filesystem (which is how we installed oprofile, obviously), but there is no way for a runtime program to write to /root, whether it is superuser or not.
I tried creating a symlink in the ROM that points /root/.oprofile -> /var/tmp/oprofile, but apparently oprofile doesn't see this symlink as a directory, and fails when run:
redacted#redacted:~$ sudo opcontrol --no-vmlinux --start --session-dir=/var/tmp/oprofile/foo
mkdir: cannot create directory `/root/.oprofile': File exists
Couldn't mkdir -p /root/.oprofile
We must run our profilers on this particular system, because the performance issues we're trying to investigate don't manifest if we build and run the app on a development server. We can't just run our tests on a programmer's workstation and profile the app there, because the problem doesn't happen there.
Is there some way to configure oprofile so that it doesn't use /root ?
I guess it should be as simple as overriding the HOME environment variable:
HOME=/tmp/fakehome sudo -E opcontrol --no-vmlinux --start --session-dir=/var/tmp/oprofile/foo
If that doesn't work out, you could have a look at
unionfs
aufs
to create a writable overlay. You might even just mount tmpfs on /root,or something simple like that.
It turns out that this directory is hardcoded into the opcontrol bash script:
# location for daemon setup information
SETUP_DIR="/root/.oprofile"
SETUP_FILE="$SETUP_DIR/daemonrc"
Editing those lines seemed to get it working, more or less.