Cassandra - migrate data folder (change data_file_directories value in config) - cassandra

We have a 3-node Cassandra cluster (version 3.16). On one of the node (10.0.4.4) we need to move the data from one drive to another drive on the same node. On SO and other web sites, we have seen the following procedure:
Start by rsyncing the data from old location to new location:
sudo rsync -avzP --delete /var/lib/cassandra/data /datadrive/cassandra/data
Repeat until it is fast enough. Then drain & flush the node
nodetool flush
nodetool drain
Stop cassandra service. We are running it on systemd so we issue
sudo systemctl stop cassandra
Run again rsync
sudo rsync -avzP --delete /var/lib/cassandra/data /datadrive/cassandra/data
Update the cassandra.yaml and set data_file_directories:
data_file_directories:
- /datadrive/cassandra/data
Ensure permissions are set:
chown -R cassandra:cassandra /datadrive/cassandra/data
Restart the node:
sudo systemctl start cassandra
When doing this sequence, at start-up we get the following error:
ERROR [main] 2020-04-18 14:51:51,742 CassandraDaemon.java:774 - Exception encountered during startup
Apr 18 14:51:51 i1 cassandra: java.lang.RuntimeException: A node with address /10.0.4.4 already exists,
cancelling join. Use cassandra.replace_address if you want to replace this node.
What are we missing? The node is not being replace, so I am hesitant to set replace_address.
What is the right way to change the data folder of a running node that already has data?
Thank you.

Well, I finally found the issue. I am posting the answer as I am thinking it may help people who makes the same mistake. The problem was this line:
sudo rsync -avzP --delete /var/lib/cassandra/data /datadrive/cassandra/data
The syntax to sync the contents of dir1 to dir2 on the same system is:
rsync -r dir1/ dir2
Notice the extra / after dir1. So the rsync command corrected is:
sudo rsync -avzP --delete /var/lib/cassandra/data/ /datadrive/cassandra/data
And once this is done, everything works perfectly...

Related

cassandra Backup from one node to another node

I'm new to cassandra and gremlin.i am using gremlin to enter and retrive the data from cassandra .i want to take a bakup and restore it on new node.i took a snapshot using nodetool.please help me with some links or documents
I used the secound approach of this post : How do I replicate a Cassandra's local node for other Cassandra's remote node?
If structure of the tables is the same, you could create two bash's scripts like below:
1. Export the data using these commands:
nodetool flush <your-keyspace-name>
nodetool cleanup <your-keyspace-name>
nodetool -h localhost -p 7199 snapshot <your-keyspace-name>
zip -r /tmp/bkp.zip /var/lib/cassandra/data/<your-keyspace-name>/
sshpass -p <password> scp -v /tmp/bkp.zip root#<ip>:/tmp
2. Import the data:
unzip /tmp/bkp.zip
nodetool cleanup <your-keyspace-name>
cd /var/lib/cassandra/data/<your-keyspace-name>/ && find /var/lib/cassandra/data/<your-keyspace-name>/ -maxdepth 5 -type d -exec sstableloader -v --nodes 127.0.0.1 {} \;
If you note some slow process, please check this another post: Cassandra's sstableloader too slow in import data
Important: You should adapt this informaction to your reallity.

zookeeper_server.pid: Permission denied FAILED TO WRITE PID

Sorry, if there are the same question. I'll do project on hbase that need to install zookeeper. So I follow the instruction on youtube : https://www.youtube.com/watch?v=ZoYteLiNHNc
Finally I want to start zkServer.sh but
ardias#Arif-Satellite-C800D:~$ zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... /usr/local/hadoop/zookeeper-3.4.9/bin/zkServer.sh: line 149: /usr/local/hadoop/zoo_data/zookeeper_server.pid: Permission denied
FAILED TO WRITE PID
I think the problem is I am not superuser so I use sudo but
ardias#Arif-Satellite-C800D:~$ sudo zkServer.sh start
[sudo] password for ardias:
sudo: zkServer.sh: command not found
I've been search any solutions on
FAILED TO WRITE PID installing Zookeeper and also
Zookeeper: FAILED TO WRITE PID
but it doesn't work on me. There's file myid in /usr/local/hadoop/zoo_data contains character "1".
Anybody have solutions?
This is my zoo.cfg and my ubuntu version is 14.04.
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/hadoop/zoo_data
clientPort=2181
server.1=localhost:2888:3888
If someone is still facing this issue on MAC, then find below
ZOO_DATADIR="$(echo -e "${ZOO_DATADIR}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
in zkServer.sh file and remove option -e from echo command.
The final command should be ZOO_DATADIR="$(echo "${ZOO_DATADIR}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
Restart zookeeper after this.
Two solutions:
1. change dataDir to a directory that ardias have permission.
2. find the absolute path of zkServer.sh and use sudo /$absolutePath/zkServer.sh start
Please check the zoo_sample.cfg
[hadoop#localhost conf]$ vi zoo_sample.cfg
[hadoop#localhost conf]$ ls
configuration.xsl log4j.properties zoo_sample.cfg
[hadoop#localhost conf]$ mv zoo_sample.cfg zoo.cfg
change the zoo_sample.cfg to zoo.cfg

Unable to start CouchDB

Just installed CouchDb using brew on mac mountain lion. Everything went well till I hit the following issue to start the server I do not know erlnag and could not analyze the dump file
`couchdb
Apache CouchDB 1.2.1 (LogLevel=info) is starting.
{"init terminating in do_boot",{{badmatch,{error,{bad_return,{{couch_app,start,[normal,["/usr/local/etc/couchdb/default.ini","/usr/local/etc/couchdb/local.ini"]]},{'EXIT',{{badmatch,{error,shutdown}},[{couch_server_sup,start_server,1,[{file,"couch_server_sup.erl"},{line,98}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,274}]}]}}}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
Crash dump was written to: erl_crash.dump
init terminating in do_boot ()`
Any help much appreciated.
I have left the configurations files as it is
Often this is due to incorrect permissions on various configuration files & directories. It can be caused by running as a sudo / root user for example.
You can try fixing this using the following, but you may need to either create/add yourself to a couchdb group, or use a different user:group combination.
sudo chown -R couchdb:couchdb /etc/couchdb /var/lib/couchdb /var/run/couchdb /var/log/couchdb
sudo chmod -R 770 /etc/couchdb /var/lib/couchdb /var/run/couchdb /var/log/couchdb
sudo find /etc/couchdb /var/lib/couchdb /var/run/couchdb /var/log/couchdb -type f | sudo xargs chmod 660
```
See the chmod section in http://wiki.apache.org/couchdb/Installing_on_OSX for more detail.
I've had this problem when attempting to load a configuration file that doesn't exist, I was starting CouchDB with the -a option to supply additional configuration, and if that file doesn't exist I get an error similar to the one reported:
$ couchdb -a /does/not/exist.ini
{"init terminating in do_boot",{{badmatch,{error,{bad_return,{{couch_app,start,[normal,["/usr/local/etc/couchdb/default.ini","/usr/local/etc/couchdb/local.ini"]]},{'EXIT',{{badmatch,{error,{error,enoent}}},[{couch_server_sup,start_server,1,[{file,"couch_server_sup.erl"},{line,56}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,269}]}]}}}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
sudo apt-get install libicu-dev
Provide Proper permissions

rsync - mkstemp failed: Permission denied (13) [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
I have the following setup to periodically rsync files from server A to server B. Server B has the rsync daemon running with the following configuration:
read only = false
use chroot = false
max connections = 4
syslog facility = local5
log file = /var/adm/rsyncd.log
munge symlinks = false
secrets file = /etc/rsyncd.secrets
numeric ids = false
transfer logging = true
log format = %h %o %f %l %b
[BACKUP]
path = /path/to/archive
auth users = someuser
From server A I am issuing the following command:
rsync -adzPvO --delete --password-file=/path/to/pwd/file/pwd.dat /dir/to/be/backedup/ someuser#192.168.100.100::BACKUP
BACKUP directory is fully read/write/execute to everyone. When I run the rsync command from server A, I see:
afile.txt
989 100% 2.60kB/s 0:00:00 (xfer#78, to-check=0/79)
for each and everyfile in the directory I wish to backup. It fails when I get to writing tmp files:
rsync: mkstemp "/.afile.txt.PZQvTe" (in BACKUP) failed: Permission denied (13)
Hours of googling later and I still can't resolve what seems to be a very simple permission issue. Advice? Thanks in advance.
Additional Information
I just noticed the following occurs at the beginning of the process:
rsync: failed to set permissions on "/." (in BACKUP): Permission denied (13)
Is it trying to set permission on "/"?
Edit
I am logged in as the user - someuser. My destination directory has full read/write/execute permission for everyone, including it's contents. In addition, the destination directory is owned by someuser and in someuser's group.
Follow up
I've found using SSH solves this
Make sure the user you're rsync'd into on the remote machine has write access to the contents of the folder AND the folder itself, as rsync tried to update the modification time on the folder itself.
Even though you got this working, I recently had a similar encounter and no SO or Google searching was of any help as they all dealt with basic permission issues wheres the solution below is somewhat of an off setting that you wouldn't even think to check in most situations.
One thing to check for with permission denied that I recently found having issues with rsync myself where permissions were exactly the same on both servers including the owner and group but rsync transfers worked one way on one server but not the other way.
It turned out the server with problems that I was getting permission denied from had SELinux enabled which in turn overrides POSIX permissions on files/folders. So even though the folder in question could have been 777 with root running, the command SELinux was enabled and would in turn overwrite those permissions which produced a "permission denied"-error from rsync.
You can run the command getenforce to see if SELinux is enabled on the machine.
In my situation I ended up just disabling SELINUX completely because it wasn't needed and already disabled on the server that was working fine and just caused problems being enabled. To disable, open /etc/selinux/config and set SELINUX=disabled. To temporarily disable you can run the command setenforce 0 which will set SELinux into a permissive state rather then enforcing state which causes it to print warnings instead of enforcing.
Rsync daemon by default uses nobody/nogroup for all modules if it is running under root user. So you either need to define params uid and gid to the user you want, or set them to root/root.
I encountered the same problem and solved it by chown the user of the destination folder. The current user does not have the permission to read, write and execute the destination folder files. Try adding the permission by chmod a+rwx <folder/file name>.
This might not suit everyone since it does not preserve the original file permissions but in my case it was not important and it solved the problem for me. rsync has an option --chmod:
--chmod This option tells rsync to apply one or more comma-separated lqchmodrq strings to the permission of the files in the transfer. The
resulting value is treated as though it was the permissions that the
sending side supplied for the file, which means that this option can
seem to have no effect on existing files if --perms is not enabled.
This forces the permissions to be what you want on all files/directories. For example:
rsync -av --chmod=Du+rwx SRC DST
would add Read, Write and Execute for the user to all transferred directories.
I had a similar issue, but in my case it was because storage has only SFTP, without ssh or rsync daemons on it. I could not change anything, bcs this server was provided by my customer.
rsync could not change the date and time for the file, some other utilites (like csync) showed me other errors: "Unable to create temporary file Clock skew detected".
If you have access to the storage-server - just install openssh-server or launch rsync as a daemon here.
In my case - I could not do this and solution was: lftp.
lftp's usage for syncronization is below:
lftp -c "open -u login,password sftp://sft.domain.tld/; mirror -c --verbose=9 -e -R -L /srs/folder /rem/folder"
/src/folder - is the folder on my PC, /rem/folder - is sftp://sft.domain.tld/rem/folder.
you may find mans by the link lftp.yar.ru/lftp-man.html
Windows: Check permissions of destination folders. Take ownership if you must to give rights to the account running the rsync service.
I had the same issue in case of CentOS 7. I went through lot of articles ,forums but couldnt find out the solution.
The problem was with SElinux. Disabling SElinux at the server end worked.
Check SELinux status at the server end (from where you are pulling data using rysnc)
Commands to check SELinux status and disable it
$getenforce
Enforcing ## this means SElinux is enabled
$setenforce 0
$getenforce
Permissive
Now try running rsync command at the client end ,it worked for me.
All the best!
I have Centos 7 server with rsyncd on board:
/etc/rsyncd.conf
[files]
path = /files
By default selinux blocks access for rsyncd to /files folder
# this sets needed context to my /files folder
sudo semanage fcontext -a -t rsync_data_t '/files(/.*)?'
sudo restorecon -Rv '/files'
# sets needed booleans
sudo setsebool -P rsync_client 1
Disabling selinux is an easy but not a good solution
I had the same issue, so I first SSH into the server to confirm that I able to log in to the server by using the command:
ssh -i /Users/Desktop/mypemfile.pem user#ec2.compute-1.amazonaws.com
Then in New Terminal
I copied a small file to the server by using SCP, to make sure I am able to make a connection:
scp -i /Users/Desktop/mypemfile.pem /Users/Desktop/test.file user#ec2.compute-1.amazonaws.com:/home/user/test/
Then In the same new terminal, I tried running rsync:
rsync -avz -e "ssh -i /Users/Desktop/mypemfile.pem" /Users/Desktop/backup/image.img.gz user#ec2.compute-1.amazonaws.com:
If you're on a Raspberry pi or other Unix systems with sudo you need to tell the remote machine where rsync and sudo programs are located.
I put in the full path to be safe.
Here's my example:
rsync --stats -paogtrh --progress --omit-dir-times --delete --rsync-path='/usr/bin/sudo /usr/bin/rsync' /mnt/drive0/ pi#192.168.10.238:/mnt/drive0/
I imagine a common error not currently mentioned above is trying to write to a mount space (e.g., /media/drivename) when the partition isn't mounted. That will produce this error as well.
If it's an encrypted drive set to auto-mount but doesn't, might be an issue of auto-unlocking the encrypted partition before attempting to write to the space where it is supposed to be mounted.
I had the same error while syncing files inside of a Docker container and the destination was a mounted volume (Docker for mac), I run rsync via su-exec <user>. I was able to resolve it by running rsync as root with -og flags (keep owner and group for destination files).
I'm still not sure what caused that issue, the destination permissions were OK (I run chown -R <user> for destination dir before rsync), perhaps somehow related to Docker for Mac slow filesystem.
Take attention on -e ssh and jenkins#localhost: in next example:
rsync -r -e ssh --chown=jenkins:admin --exclude .git --exclude Jenkinsfile --delete ./ jenkins#localhost:/home/admin/web/xxx/public
That helped me
P.S. Today, i realized that when you change (add) jenkins user to some group, permission will apply after slave (agent) restart. And my solution (-e ssh and jenkins#localhost:) need only when you can't restart agent/server.
Yet still another way to get this symptom: I was rsync'ing from a remote machine over ssh to a Linux box with an NTFS-3G (FUSE) filesystem. Originally the filesystem was mounted at boot time and thus owned by root, and I was getting this error message when I did an rsync push from the remote machine. Then, as the user to which the rsync is pushed, I did:
$ sudo umount /shared
$ mount /shared
and the error messages went away.
The group user name for the destination directory and sub directories should be same as per the user.
if the user is 'abc' then the destination directory should be
lrwxrwxrwx 1 abc abc 34 Jul 18 14:05 Destination_directory
command chown abc:abc Destination_directory
Surprisingly nobody have mentioned all powerful SUDO.
Had the same problem and sudo fixed it
run in root access ssh chould solve this problem
or chmod 0777 /dir/to/be/backedup/
or chown username:user /dir/to/be/backedup/

rsync over SSH preserve ownership only for www-data owned files

I am using rsync to replicate a web folder structure from a local server to a remote server. Both servers are ubuntu linux. I use the following command, and it works well:
rsync -az /var/www/ user#10.1.1.1:/var/www/
The usernames for the local system and the remote system are different. From what I have read it may not be possible to preserve all file and folder owners and groups. That is OK, but I would like to preserve owners and groups just for the www-data user, which does exist on both servers.
Is this possible? If so, how would I go about doing that?
** EDIT **
There is some mention of rsync being able to preserve ownership and groups on remote file syncs here: http://lists.samba.org/archive/rsync/2005-August/013203.html
** EDIT 2 **
I ended up getting the desired affect thanks to many of the helpful comments and answers here. Assuming the IP of the source machine is 10.1.1.2 and the IP of the destination machine is 10.1.1.1. I can use this line from the destination machine:
sudo rsync -az user#10.1.1.2:/var/www/ /var/www/
This preserves the ownership and groups of the files that have a common user name, like www-data. Note that using rsync without sudo does not preserve these permissions.
You can also sudo the rsync on the target host by using the --rsync-path option:
# rsync -av --rsync-path="sudo rsync" /path/to/files user#targethost:/path
This lets you authenticate as user on targethost, but still get privileged write permission through sudo. You'll have to modify your sudoers file on the target host to avoid sudo's request for your password. man sudoers or run sudo visudo for instructions and samples.
You mention that you'd like to retain the ownership of files owned by www-data, but not other files. If this is really true, then you may be out of luck unless you implement chown or a second run of rsync to update permissions. There is no way to tell rsync to preserve ownership for just one user.
That said, you should read about rsync's --files-from option.
rsync -av /path/to/files user#targethost:/path
find /path/to/files -user www-data -print | \
rsync -av --files-from=- --rsync-path="sudo rsync" /path/to/files user#targethost:/path
I haven't tested this, so I'm not sure exactly how piping find's output into --files-from=- will work. You'll undoubtedly need to experiment.
As far as I know, you cannot chown files to somebody else than you, if you are not root. So you would have to rsync using the www-data account, as all files will be created with the specified user as owner. So you need to chown the files afterwards.
The root users for the local system and the remote system are different.
What does this mean? The root user is uid 0. How are they different?
Any user with read permission to the directories you want to copy can determine what usernames own what files. Only root can change the ownership of files being written.
You're currently running the command on the source machine, which restricts your writes to the permissions associated with user#10.1.1.1. Instead, you can try to run the command as root on the target machine. Your read access on the source machine isn't an issue.
So on the target machine (10.1.1.1), assuming the source is 10.1.1.2:
# rsync -az user#10.1.1.2:/var/www/ /var/www/
Make sure your groups match on both machines.
Also, set up access to user#10.1.1.2 using a DSA or RSA key, so that you can avoid having passwords floating around. For example, as root on your target machine, run:
# ssh-keygen -d
Then take the contents of the file /root/.ssh/id_dsa.pub and add it to ~user/.ssh/authorized_keys on the source machine. You can ssh user#10.1.1.2 as root from the target machine to see if it works. If you get a password prompt, check your error log to see why the key isn't working.
I had a similar problem and cheated the rsync command,
rsync -avz --delete root#x.x.x.x:/home//domains/site/public_html/ /home/domains2/public_html && chown -R wwwusr:wwwgrp /home/domains2/public_html/
the && runs the chown against the folder when the rsync completes successfully (1x '&' would run the chown regardless of the rsync completion status)
Well, you could skip the challenges of rsync altogether, and just do this through a tar tunnel.
sudo tar zcf - /path/to/files | \
ssh user#remotehost "cd /some/path; sudo tar zxf -"
You'll need to set up your SSH keys as Graham described.
Note that this handles full directory copies, not incremental updates like rsync.
The idea here is that:
you tar up your directory,
instead of creating a tar file, you send the tar output to stdout,
that stdout is piped through an SSH command to a receiving tar on the other host,
but that receiving tar is run by sudo, so it has privileged write access to set usernames.
rsync version 3.1.2
I mostly use windows in local, so this is the command line i use to sync files with the server (debian) :
user#user-PC /cygdrive/c/wamp64/www/projects
$ rsync -rptgoDvhP --chown=www-data:www-data --exclude=.env --exclude=vendor --exclude=node_modules --exclude=.git --exclude=tests --exclude=.phpintel --exclude=storage ./website/ username#hostname:/var/www/html/website

Resources