I am starting to work with kudu and the only way to measure the size of a table in kudu is throw the Cloudera Manager - KUDU - Chart Library - Total Tablet Size On Disk Across Kudu Replicas.
There are another way to know it throw command line?
You can run the below query to get a fair idea of the size of the tables.
CM->Chart builder select total_kudu_on_disk_size_across_kudu_replicas where category=KUDU_TABLE
I find a way to know the size of a kudu table throw the cloudera API.
date=`date +%F`
curl -u $user --ssl --cacert ca-truststore.pem "https://$node:7183/api/v18/timeseries?query=select+total_kudu_on_disk_size_across_kudu_replicas+where+category%3DKUDU_TABLE&contentType=text%2Fcsv&from=$date"
In that case the output is in csv format.
If you want a json format:
curl -u $user --ssl --cacert ca-truststore.pem "https://$node:7183/api/v18/timeseries?query=select+total_kudu_on_disk_size_across_kudu_replicas+where+category%3DKUDU_TABLE&contentType=application%2Fjson&from=$date"
Related
I need to transfer around 50 databases from one server to other. Tried two different backup options.
1. pg_dump -h XX.XXX.XX.XXX -p pppp -U username -Fc -d db -f db.cust -- all backups in backup1 folder size 10GB
2. pg_dump -h XX.XXX.XX.XXX -p pppp -U username -j 4 -Fd -d db -f db.dir -- all backups in backup2 folder size 10GB
then transferred to other server for restoration using scp.
scp -r backup1 postgres#YY.YYYY.YY.YYYY:/backup/
scp -r backup2 postgres#YY.YYYY.YY.YYYY:/backup/
Noticed a strange thing. Though the backup folder size are same for both backup but it takes different time to transfer using scp. For directory format backup, transfer is 4 times than custom format backup. Both the SCP done in same network and tried multiple times but result are some.also tried rsync but no difference.
please suggest, what could be the reason for slowness and how can I speed up. I am open to use any other method to transfer.
Thanks
I'm new to cassandra and gremlin.i am using gremlin to enter and retrive the data from cassandra .i want to take a bakup and restore it on new node.i took a snapshot using nodetool.please help me with some links or documents
I used the secound approach of this post : How do I replicate a Cassandra's local node for other Cassandra's remote node?
If structure of the tables is the same, you could create two bash's scripts like below:
1. Export the data using these commands:
nodetool flush <your-keyspace-name>
nodetool cleanup <your-keyspace-name>
nodetool -h localhost -p 7199 snapshot <your-keyspace-name>
zip -r /tmp/bkp.zip /var/lib/cassandra/data/<your-keyspace-name>/
sshpass -p <password> scp -v /tmp/bkp.zip root#<ip>:/tmp
2. Import the data:
unzip /tmp/bkp.zip
nodetool cleanup <your-keyspace-name>
cd /var/lib/cassandra/data/<your-keyspace-name>/ && find /var/lib/cassandra/data/<your-keyspace-name>/ -maxdepth 5 -type d -exec sstableloader -v --nodes 127.0.0.1 {} \;
If you note some slow process, please check this another post: Cassandra's sstableloader too slow in import data
Important: You should adapt this informaction to your reallity.
I need to untar hundreds of tar files of a size between 30 and 140GB as soon as they become available. I've created a pool with (currently) 5 "standard a1" single core Linux nodes, " microsoft-azure-batch centos-container 7-4 (latest)"
The batch has an associated storage account. The storage contains the tar files which I wish to untar into that same store. There is also a job called, "untarjob"
Since the batch allows for an association to a store I made an assumption that would be naturally available to each compute node. However, the assumption now is perhaps that isn't true? As such I'm trying to mount the store and run the unix tar command to untar as shown below.
This isn't working:
az batch task create --task-id t80000 --job-id untarjob --command-
line "mount -t cifs //dummy.file.core.windows.net/s01 /tarfiles -o
vers=3.0,username=dummy,password=Pv4<snipp>N8Imu34reseEEe==,dir_mode=0777,
file_mode=0777,sec=ntlmssp ; tar -xvf /tarfiles/80000.tar"
Could you tell me how I could solve this please?
Many thanks
Kevin
First, you probably shouldn't use a container-specific VM image if your tasks don't need containers and need to run directly on the host.
For the issue, wrap your command in a shell invocation:
/bin/sh -c 'sudo mkdir /tarfiles; sudo mount ...; tar ...'
Note the single quotes since you're using double quotes for the --command-line arg. Also, if your command needs to run privileged entirely, consider using user identities to avoid having to use sudo.
I want to use mysqldump to get a table from a remote MySQL server, then compress this downloaded file using zip. While downloading, I wish I can view the progress using pv.
Can I do the above things with | in one line of command?
These are what I've tried:
mysqldump -uuser_name -ppassword -hremote_address --routines my_database my_table | pv | zip > my_database_my_table.sql.zip
The problem with this command is that when executing unzip my_database_my_table.sql.zip, I got - as the name of the output file. I wish I could determine the file's name when I execute zip command.
Is it possible to set the name of the inflated file?
mysqldump -uuser_name -ppassword -hremote_address --routines my_database my_table | pv | zip my_database_my_table.sql > my_database_my_table.sql.zip
This command gives me mysqldump: Got errno 32 on write error.
Here is how I used them together:
mysqldump -u db_user -pdb_password db_name| pv | zip > backup.zip
Here is https://stackoverflow.com/a/50985546/3778130 full automatic backup script if someone is interested.
Hope it helps someone down the road.
A quick man zip shows that you can use -O or --output-file to specify the output name. That should do you.
In order to be able to get the progress out of pv, it needs to know the full size of the data. Unfortunately, by piping it to mysqldump it does not have that kind of data.
I would recommend to dump the data first and transfer and restore it later.
Dump it.
pv mysqldump.sql | zip > ~/mysqldump.sql.zip
scp remotehost:/home/folder/mysqldump.sql ./
unzip it
pv mysqldump.sql | mysql -u<dbuser> -p <dbname>
I am not familiar with pv, but my zip documentation states for zip to use the stdin, the dash must be specified. So i use this :
mysqldump --user=username --password=password mydatabase | zip mydatabase.zip -
What would be the most efficient way for me to export 200 databases with a total of 40GB of data, and import them into another server? I was originally planning on running a script that would export each DB to their own sql file, and then import them into the new server. If this is the best way, are there some additional flags i can pass to the mysqldump that will speed it up?
The other option I saw was to directly pipe the mysqldump into an import over SSH. Would this be a better option? If so could you provide some info on what the script might look like?
If the servers can ping each other you could use PIPES to do so:
mysqldump -hHOST_FROM -u -p db-name | mysql -hHOST_TO -u -p db-name
Straightforward!
[EDIT]
Answer for your question:
mysqldump -hHOST_FROM -u -p --all | mysql -hHOST_TO -u -p
The quick and fastest way is to use percona xtrabackup to take hot backups. It is very fast and you can use it on live system whereas mysqldump can cause locking. Please avoid copying /var/lib directory to other server in case of Innodb, this would have very bad effects.
Try percona xtrabackup, here is some more information on this on installation and configuration. Link here.
If both mysql servers will have same dbs and config I think the best method is to copy the /var/lib/mysql dir using rsync. Stop servers before doing the copy to avoid table corruption
Export MySQL database using SSH with the command
mysqldump -p -u username database_name > dbname.sql
Move the dump using wget from the new server SSH.
wget http://www.domainname.com/dbname.sql
Import the MySQL database using SSH with the command
mysql -p -u username database_name < file.sql
Done!!