In Cassandra, is there a way to generate CREATE TABLE statements for all the existing tables inside a particular keyspace?
DESC KEYSPACE KEYSPACE_NAME
Output CQL commands for the given keyspace. These CQL commands can be used to recreate the keyspace and tables.
Related
I created a hive table on top of a parquet folder written via spark. In one test server it is running fine and giving out results (hive version 2.6.5.196) but in production it gives no records (hive 2.6.5.179). Could someone please point out what the exact issue could be?
If you created the table on top of an existing partition structure, you have to make it known to the table that there are partitions at this location.
MSCK REPAIR TABLE table_name; -- adds missing partitions
SELECT * FROM table_name; -- should return records now
This problem shouldn't happen if there are only files in that location, and if they are the expected format.
You can verify with:
SHOW CREATE TABLE table_name; -- to see the expected format
created hive table on top of a parquet folder written via spark.
Check for the databases that you are using is available or not using
show databases;
check the ddl of the table that you have created on your test server and the other that is there on production
show create table table_name;
Make sure that both the ddl exactly matches.
Do msck repair table table_name to load the incremental data or the data from all the partitions
select * from table_name to view records
For Example: I want to create 40 tables in one keyspace. In 40 tables I want to shard 3 tables. Is is it possible to shard specific tables without creating new keyspace.
I have seen How to shard only specific tables using vitess But for this we need to create new keyspace. I don't want to create new keyspace. I want sharded and unsharded tables in one keyspace is it possible?
This is currently not possible. A keyspace is categorized as sharded or unsharded. So, you have to migrate the tables you want to shard into a sharded keyspace and then reshard the keyspace.
Some people worked around this by assigning a "null primary vindex" to the unsharded tables, essentially forcing all rows to live in the first shard. But I don't know if this was experimental or was actually used in production.
I have a external table that has a partitioned column called rundate. I can load data into the table using
DataFrame.write.mode(SaveMode.Overwrite).orc("s3://test/table")
I then create a partition using
spark.sql("ALTER TABLE table ADD IF NOT EXISTS PARTITION(rundate = '2017-12-19')")
The code works fine and i can see the partitions. But I cannot see data in the Hive table.
You have not saved the partition data in correct folder structure and also manually added the partition where data does not exist.
Two things:
1. First make sure you are saving at data at the location where external table is created and also the folder structure is same as hive expect. e.g Assume your external table name is table and partition column is rundate, partition value is 2017-12-19 and external table is pointing to location s3://test/table. Then save data for partition 2017-12-19 as below:
DataFrame.write.mode(SaveMode.Overwrite).orc("s3://test/table/rundate=2017-12-19/")
2.Once save is successful below command to update the metastore of hive with the latest added partition.
synatx: msck repair table <tablename>
msck repair table table
I want to migrate 500 tables from mysql to cassandra but do not want to create the schemas in cassandra before migration.
i know the option of CQL-IMPORT in Sqoop but only allows copying data with tables created in cassandra.
Is there any way where i can have all the tables structure copied from MYSQL to Cassandra schema format creation of 500 tables in cassandra with more than 100 columns per table will be time consuming.
please help
I created an old keyspace in Cassandra cluster but found the definition of its "comparator" is wrong, so I have to recreate a new keyspace and do data migration. Is there any tool to do data migration? or I have to program with Thrift client read all data from old keyspace and write them to new keyspace? Any suggestions or code snippets is welcome!
This is a commun question, and I think it has been asked before here.
You can use the COPY command in C*.
You will find more details here http://www.datastax.com/dev/blog/ways-to-move-data-tofrom-datastax-enterprise-and-cassandra
We can do it using COPY command in cql. Using COPY command we can save table data to .csv file and back to a table from .csv file. But, the better approach will be to write a program to read from table and write it to another table because importing from csv may fail if the table contains collection column types like list<text>, map<text, text>, set<text>.
Eg :-
To copy table data from table to .csv file :-
COPY keyspace1.table1 (column1, column2) TO 'path/to/file/keyspace1_table1.csv';
To copy csv data from file to a table :-
COPY keyspace2.table1 (column1, column2) FROM 'path/to/file/keyspace1_table1.csv';
Refer Cassandra migration tool