I am using Jconsole for monitoring Cassandra. I can get value like how much load each keyspace is having.
I want to find out disk space usage for each node in a cluster by remotely.
Is there any way to do so?
A shell script can do the trick
for i in node1_ip node2_ip ... nodeN_ip
do
ssh user#$i "du -sh /var/lib/cassandra/data" >> /tmp/disk_usage.txt
done
Replace /var/lib/cassandra/data if your data folder is put somewhere else
Related
I have been looking for a solution to setup an environment variable, that is non-persistent after reboot, but still accessable system wide. Anyone with a solution (bash)?
The reason is that I want crontab etc. to use these variables (after they have been set once in a single session). But, if the power goes or the disk is taken physically the variable (a password) is not found in the file system. (It's a Raspberry Pi backup server).
So far I have considered using Screen (with no real luck), and is considering a script in /etc/profile.d
The reason is that I want crontab etc. to use these variables (after they have been set once in a single session). But, if the power goes or the disk is taken physically the variable
So store the password in memory, assuming that memory is cleared after power loss. Ex. use shared memory or tmpfs filesystem. First script to setup the password:
#!/bin/bash
# setup_password.sh
if ! findmnt /srv/mypassword >/dev/null; then
mount -o tmpfs tmpfs /srv/mypassword
fi
printf "%s\n" "$1" > /srv/mypassword/password.txt
and script you execute from cron:
#!/bin/bash
if ! password=$(cat /srv/mypassword/password.txt); then
echo "ERROR: password not setup" >&2
exit 2
fi
: continue and use the password "$password"
If your /tmp is a tmpfs, and probably it is, just store the password in /tmp...
I want to take Cassandra backup at every 1hr interval and move it to Shared location.
Cassandra taking the snapshot in the default location, how can I take the snapshot on /opt/backup location?
You can't (with snapshots).
nodetool snapshot -t <tag> <keyspace> is a quite simple tool - it just creates hard links for every file in your keyspace data directories to snapshots/<tag>.
Since these are hard links they have to be on the same filesystem. Benefit of those hard links is that a snapshot is quite fast and doesn't consume additional disk space initially (when sstables got compacted / deleted the files remain in the snapshot).
If you want those backups in a different location use -t <tag> while creating your snapshot. I made up a demo with demosnapshot and a simple script (not fully elaborated but shows the idea:
$ cat cassandrabackup.sh
#!/bin/bash
TAG=`date +%Y%m%d%H%M%S`
BACKUP_LOC=/tmp/backup/`hostname`
KEYSPACE=demokeyspace
echo creating snapshot $TAG
nodetool snapshot -t $TAG $KEYSPACE
echo sync to backup location $BACKUP_LOC
find /var/lib/cassandra -type f -path "*snapshots/$TAG*" -printf %P\\0 | rsync -avP --files-from=- --from0 /var/lib/cassandra/ $BACKUP_LOC
echo removing snapshot $TAG
nodetool clearsnapshot -t $TAG
The script creates a snaphot with a specific tag (datetime), rsyncs the contents to a backup location and then removes the snapshot. If KEYSPACE is not defined all keyspaces are backuped.
Result is like this:
$ ./cassandrabackup.sh
creating snapshot 20170823132936
Requested creating snapshot(s) for [demokeyspace] with snapshot name [20170823132936] and options {skipFlush=false}
Snapshot directory: 20170823132936
sync to backup location /tmp/backup/host1.domain.tld
building file list ...
6 files to consider
data1/
data1/demokeyspace/
data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/
data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/
data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823132936/
data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823132936/manifest.json
13 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/6)
sent 305 bytes received 50 bytes 710.00 bytes/sec
total size is 13 speedup is 0.04
removing snapshot 20170823132936
Requested clearing snapshot(s) for [all keyspaces] with snapshot name [20170823132936]
$ ifjke#fsca01:~$ find /tmp/backup/
/tmp/backup/
/tmp/backup/host1.domain.tld
/tmp/backup/host1.domain.tld/data2
/tmp/backup/host1.domain.tld/data2/demokeyspace
/tmp/backup/host1.domain.tld/data2/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823
/tmp/backup/host1.domain.tld/data2/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots
/tmp/backup/host1.domain.tld/data2/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823125951
/tmp/backup/host1.domain.tld/data2/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823125951/manifest.json
/tmp/backup/host1.domain.tld/data2/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823130014
/tmp/backup/host1.domain.tld/data2/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823130014/manifest.json
/tmp/backup/host1.domain.tld/data1
/tmp/backup/host1.domain.tld/data1/demokeyspace
/tmp/backup/host1.domain.tld/data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823
/tmp/backup/host1.domain.tld/data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots
/tmp/backup/host1.domain.tld/data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823132936
/tmp/backup/host1.domain.tld/data1/demokeyspace/demotable-0bbb579087ef11e7aa786377cd3ba823/snapshots/20170823132936/manifest.json
$
As I did that error by myself in the past - include hostname in the backups ;)
Apart from that there is also an incremental backup feature in cassandra:
http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsBackupIncremental.html
I’m a beginner programmer and I'm try to learn how to successfully mount a disk image and analyse it but can't fine any guides online or any mention on web pages.
I’ve set myself the task as I’m thinking of joining a computer forensics course next year and believe these skills will give me a head start.
This is the code I've made so far but I've become stuck. I want the script to be able to extract command history data for all users, and also log successful and unsuccessful login attempts from log files such as /var/log/wtmp.
I’m not exactly looking for someone to complete the code (as that would be counterproductive) but to point me towards hints and tips, guides and tutorials to get over these early stage of programming.
#!/bin/bash
mount="/myfilesystem"
if grep -qs "$mount" /proc/mounts; then
echo "It's mounted."
else
echo "It's not mounted."
mount "$mount"
if [ $? -eq 0 ]; then
echo "Mount success!"
else
echo "Something went wrong with the mount..."
fi
fi
sudo fdisk -l | grep/bin /sbin
For mounting a filesystem, you need two arguments at least.
The image file or block device to be mounted and
The place where to mount it in your filesystem
So, if you want to mount some external USB drive, that e.g. shows as /dev/sda and has a single partition (sda1), you need to do the following:
Find or create a directory to mount your device (easiest as root), say you created a directory /root/mountpoint
Execute the mount command: mount /dev/sda1 /root/mountpoint
You then can step into the mounted filesystem cd /root/mountpoint and look around.
Just as a sidenote: For forensics, you should always draw an image from the device (e.g. dd if=/dev/sda1 of=/root/disk.img) to avoid destroying any evidence and then mount it through the loop driver (losetup /dev/loop1 /root/disk.img; mount /dev/loop1 /root/mountpoint).
Hope this gives you a hint to start over...
When I run nodetool clearsnapshot I get the normal "Requested clearing snapshot(s)" message, but the snapshot is never removed. What can I do to troubleshoot why this is occurring? Is it acceptable for me to just manually remove the snapshot directories from the tablespace directories as a workaround for this?
nodetool clearsnapshot 1472489985541
Requested clearing snapshot(s) for [1472489985541]
nodetool listsnapshots | awk '{print $1}' | grep ^1 | sort -u
1472489985541
1473165734236
1473690660090
1474296554367
Is it acceptable for me to just manually remove the snapshot directories from the tablespace directories as a workaround for this?
Yes, you can always safely remove the snapshots directories manually. They are just hard links to actual SSTables
In order to delete a snapshot from all keyspaces using the snapshot name, you must specify the -t flag in your clearsnapshot command.
I am a Windows admin and dev, I do not generally work with Linux so forgive me if this is in some way obvious.
I have a not so good Linux box, some older version of Open SUSE, and I have a script that unmounts the USB thumb drive, formats it, and then waits for the device to become ready again before it runs a script that does a copy/MD5 checksum verification on the source and destination file to ensure the copy was valid. The problem is that on one box the USB thumb drive does not become ready after the format in a consistent way. It takes anywhere from 1 to 2+ minutes before I can access the drive via /media/LABELNAME.
The direct path is /dev/sdb but, of course, I cannot access it directly via this path to copy the files. Here is my shell script as it stands:
#!/bin/bash
set -e
echo "Starting LABELNAME.\n\nUnmounting /dev/sdb/"
umount /dev/sdb
echo "Formatting /dev/sdb/"
mkfs.vfat -I -F32 -n "LABELNAME" /dev/sdb
echo "Waiting on remount..."
sleep 30
echo "Format complete. Running make master."
perl /home/labelname_master.20120830.pl
Any suggestions? How might I wait for the drive to become ready and detect it? I have seen Detecting and Writing to a USB Key / Thumb DriveAutomatically but quite frankly I don't even know what that answer means.
It seems that you have some automatic mounting service running which detects the flash disk and mounts the partition. However, you already know what the partition is, so I recommend that you simply mount the disk in your script, choosing a suitable mount point yourself.
mkfs.vfat -I -F32 -n "LABELNAME" /dev/sdb
echo "Format complete, remounting"
mount /dev/sdb $mountpoint #<-- you would choose $mountpoint
echo "Running make master."
perl /home/labelname_master.20120830.pl