I have inserted string and integer values into dynamic columns in a Cassandra Column Family. When I query for the values in CQL they are displayed as hex encoded bits.
Can I somehow tell the query to decode the value into a string or integer?
I also would be happy to do this in the CLI if that's easier. There I see you can specify assume <column_family> validator as <type>;, but that applies to all columns and they have different types, so I have to run the assumption and query many times.
(Note that the columns are dynamic, so I haven't specified the validator when creating the column family).
You can use ASSUME in cqlsh like in cassandra-cli (although it only applies to printing values, not sending them, but that ought to be ok for you). You can also use it on a per-column basis, like:
ASSUME <column_family> ('anchor:cnnsi.com') VALUES ARE text;
..although (a), I just tested it, and this functionality is broken in cassandra-1.1.1 and later. I posted a fix at CASSANDRA-4352. And (b), this probably isn't a very versatile or helpful solution for more than a few one-off uses. I'd strongly recommend using CQL 3 here, as CQL direct support for wide storage engine rows like this is deprecated. Your table here is certainly adaptable to an (easier to use) CQL 3 model, but I couldn't say exactly what it would be without knowing more about how you're using it.
Related
Say I have 2 rows of time-series data that have exactly the same timestamp etc for the primary keys. The only difference is the rest of the data are different.
So if set 1 has [timestamp, other keys...], value_col1, value_col2, then the value_col1 and value_col2 for set 2 will have different values than set 1.
Now if I put those sets into one batch to be inserted or very quickly using seperate inserts queries, the result I see in the database can be somewhat inconsistent: it can be that value_col1 from set 1 is combined with that of value_col2 in the final row.
It took me the whole evening to find out this is actually a bug (or maybe intended behaviour...) I have my workaround now using a slightly increased timestamp for set 2. The sympton won't be noticed in many cases, but in my case where col1 is the partial decoding key for col2, then I have a problem!
Does anyone have the same problem or knows where the problem actually lies?
I'm using cassandra-node drive on nodejs 5.0.0 with cassandra 2.0.14.
You should define your in Cassandra schema to avoid race conditions.
If you provide different data with same partition and clustering keys in the same instant, you will not be able to define which should be preserved.
If you want the 2 rows to be preserved, you should use a uuid or timeuuid datatypes.
See more information in the nodejs driver docs: http://docs.datastax.com/en/developer/nodejs-driver/2.2/nodejs-driver/reference/uuids-timeuuids.html
The common query building pattern in HiveQL (and SQL in general) is to either select all columns (SELECT *) or an explicitly-specified set of columns (SELECT A, B, C). SQL has no built-in mechanism for selecting all but a specified set of columns.
There are various mechanisms for excluding some columns as outlined in this SO question but none apply naturally to HiveQL. (For example, the idea to create a temporary table with SELECT * then ALTER TABLE DROP some of its columns would wreak havoc in a big data environment.)
Ignoring the ideological discussion about whether it is a good idea to select all but some columns, this question is about the possible ways to extend Hive with this capability.
Prior to Hive 0.13.0 SELECT could take regular-expression-based columns, e.g., property_.* inside a backtick-quoted string. #invoketheshell's answer below refers to this capability but it comes at a cost, which is that, when this capability is on, Hive cannot accept columns with non-standard characters in them, e.g., $foo or x/y. That's why the Hive developers turned this behavior off by default in 0.13.0. I am looking for a generic solution that works for any column name.
A generic table-generating UDF (UDTF) could certainly do this because it can manipulate the schema. Since we are not going to generate new rows, is there a way to solve this problem using a simple row-based UDF?
This seems like a common problem with many posts around the Web showing how to solve it for various databases yet I haven't been able to find a solution for Hive. Is there code somewhere that does this?
You can choose every column except those listed in a regex based specification. This is query columns by exclusion. See below:
A SELECT statement can take regex-based column specification in Hive releases prior to 0.13.0, or in 0.13.0 and later releases if the configuration property hive.support.quoted.identifiers is set to none.
That being said you could create a new table or view using the following, and all the columns except the columns specified will be returned:
hive.support.quoted.identifiers=none;
drop table if exists database.table_name;
create table if not exists database.table_name as
select `(column_to_remove_1|...|column_to_remove_N)?+.+`
from database.some_table
where
--...
;
This will create a table that has all the columns from some_table except the columns named column_to_remove_1, ... , to column_to_remove_N. You can also choose to create a view instead.
I've been given the task of modelling a simple in Cassandra. Coming from an almost solely SQL background, though, I'm having a bit of trouble figuring it out.
Basically, we have a list of feeds that we're listening to that update periodically. This can be in RSS, JSON, ATOM, XML, etc (depending on the feed).
What we want to do is periodically check for new items in each feed, convert the data into a few formats (i.e. JSON and RSS) and store that in a Cassandra store.
So, in an RBDMS, the structure would be something akin to:
Feed:
feedId
name
URL
FeedItem:
feedItemId
feedId
title
json
rss
created_time
I'm confused as to how to model that data in Cassandra to facilitate simple things such as getting x amount of items for a specific feed in descending created order (which is probably the most common query).
I've heard of one strategy that mentions having a composite key storing, in this example, the the created_time as a time-based UUID with the feed item ID but I'm still a little confused.
For example, lets say I have a series of rows whose key is basically the feedId. Inside each row, I store a range of columns as mentioned above. The question is, where does the actual data go (i.e. JSON, RSS, title)? Would I have to store all the data for that 'record' as the column value?
I think I'm confusing wide rows and narrow (short?) rows as I like the idea of the composite key but I also want to store other data with each record and I'm not sure how to meld the two together...
You can store everything in one column family. However If the data for each FeedItem is very large, you can split the data for each FeedItem into another column family.
For example, you can have 1 column familyfor Feed, and the columns of that key are FeedItem ids, something like,
Feeds # column family
FeedId1 #key
time-stamp-1-feed-item-id1 #columns have no value, or values are enough info
time-stamp-2-feed-item-id2 #to show summary info in a results list
The Feeds column allows you to quickly get the last N items from a feed, but querying for the last N items of a Feed doesn't require fetching all the data for each FeedItem, either nothing is fetched, or just a summary.
Then you can use another column family to store the actual FeedItem data,
FeedItems # column family
feed-item-id1 # key
rss # 1 column for each field of a FeedItem
title #
...
Using CQL should be easier to understand to you as per your SQL background.
Cassandra (and NoSQL in general) is very fast and you don't have real benefits from using a related table for feeds, and anyway you will not be capable of doing JOINs. Obviously you can still create two tables if that's comfortable for you, but you will have to manage linking data inside your application code.
You can use something like:
CREATE TABLE FeedItem (
feedItemId ascii PRIMARY KEY,
feedId ascii,
feedName ascii,
feedURL ascii,
title ascii,
json ascii,
rss ascii,
created_time ascii );
Here I used ascii fields for everything. You can choose to use different data types for feedItemId or created_time, and available data types can be found here, and depending on which languages and client you are using it can be transparent or require some more work to make them works.
You may want to add some secondary indexes. For example, if you want to search for feeds items from a specific feedId, something like:
SELECT * FROM FeedItem where feedId = '123';
To create the index:
CREATE INDEX FeedItem_feedId ON FeedItem (feedId);
Sorting / Ordering, alas, it's not something easy in Cassandra. Maybe reading here and here can give you some clues where to start looking for, and also that's really depending on the cassandra version you're going to use.
So I've defined a column family that uses composite ids for the row keys. So say the composite key is CompositeType(LongType,LongType). So I've tested storing items with this type and that works fine and SELECT works as expected too when I know the full key. But lets say I want all keys that have 0 as the first element and anything as the second. So far the only way that I can see to perform this query is as follows:
if I was all keys that are 0:* then I would do a CQL query for key >= 0:0 AND key < 1:0 which works as long as there is an order preserving partitioner.
My questions are:
1) is this odd syntax only because I'm using a CQL driver (only option for nodejs aside from thrift)
2) is there any inefficiency with this type of query? essentially i'm using a composite key instead of super columns since those aren't supported in CQL. I have no problem dealing with this logic in the code as long as there is no limitations to using it like this.
I would suggest you change your data model. Use RandomPartitioner and just have the first component as the row key. Push the second component into the column names, that is make your column names composites instead.
Since column names are always sorted, you can do easy slicing operations. For example,
a) When you know both the components, do a get slice on the row key(first component) and first component of the composite.
b) When you know just the first component, fetch the complete row for the row key(first component)
This is the approach CQL3 takes when you ask it to create a table with multiple primary keys.
Your best option is to use CQL 3. This will let you use composites underneath to optimize your lookups while still allowing you to use the parts of the composite values as though they were separate columns. You're currently using composites in your row keys, and CQL 3 only supports composites in column names (so far), but that's probably ok. In many cases like this, shifting the compositing from the row key to the column name won't have an adverse effect on your performance or data distribution, but if your row keys aren't sufficiently selective, then it might.
Either way, though, you should be looking at CQL 3. CQL 2 is deprecated. I could tell you more about how to adapt your model for CQL 3 if I knew more about your situation.
I am using this as a resource to get me started - http://www.pantz.org/software/sqlite/sqlite_commands_and_general_usage.html
Currently I am working on creating an AIR program making use of the built in SQLite database. I could be considered a complete noob in making SQL queries.
table column types
I have a rather large excel file (14K rows) that I have exported to a CSV file. It has 65 columns of varying data types (mostly ints, floats and short strings, MAYBE a few bools). I have no idea about the proper form of importing so as to preserve the column structure nor do I know the best data formats to choose per db column. I could use some input on this.
table creation utils
Is there a util that can read an XLS file and based on the column headers, generate a quick query statement to ease the pain of making the query manually? I saw this post but it seems geared towards a preexisting CSV file and makes use of python (something I am also a noob at)
Thank you in advance for your time.
J
SQLite3's column types basically boil down to:
TEXT
NUMERIC (REAL, FLOAT)
INTEGER (the various lengths of integer; but INT will normally do)
BLOB (binary objects)
Generally in a CSV file you will encounter strings (TEXT), decimal numbers (FLOAT), and integers (INT). If performance isn't critical, those are pretty much the only three column types you need. (CHAR(80) is smaller on disk than TEXT but for a few thousand rows it's not so much of an issue.)
As far as putting data into the columns is concerned, SQLite3 uses type coercion to convert the input data type to the column type whereever the conversion makes sense. So all you have to do is specify the correct column type, and SQLite will take care of storing it in the correct way.
For example the number -1230.00, the string "-1230.00", and the string "-1.23e3" will all coerce to the number 1230 when stored in a FLOAT column.
Note that if SQLite3 can't apply a meaningful type conversion, it will just store the original data without attempting to convert it at all. SQLite3 is quite happy to insert "Hello World!" into a FLOAT column. This is usually a Bad Thing.
See the SQLite3 documentation on column types and conversion for gems such as:
Type Affinity
In order to maximize compatibility between SQLite and other database
engines, SQLite supports the concept of "type affinity" on columns.
The type affinity of a column is the recommended type for data stored
in that column. The important idea here is that the type is
recommended, not required. Any column can still store any type of
data. It is just that some columns, given the choice, will prefer to
use one storage class over another. The preferred storage class for a
column is called its "affinity".