Full-Text search in Cassandra;
I am fairly new to Cassandra, and wish to understand it more properly. I am attempting to perform a Full-Text search in Cassandra, but after some research I have found that there may not be a "simple" approach for this.. and I say maybe because the first page of Google hasn't said much of anything.
So I am trying to understand now instead, what is the best approach here.. This sort of lead me to take make up my own assumptions based on what I've learned so far about Cassandra, that is based on these two principals; a) design your tables based on your queries, rather than the data, and b) more-data is a good thing, as long as it is being used properly.
With that being said, I've come up with a couple of solutions I'd like to share, and also ask that if anyone has a better idea, please fill me on it before I commit to anything unreasonable/naive.
First Solution: Create a Column Family(CF), with two primary keys and an Index like so:
CREATE TABLE "FullTextSearch" (
"PartialText" text,
"TargetIdentifier" uuid,
"CompleteText" text,
"Type" int,
PRIMARY KEY ("PartialText","TargetIdentifier")
);
CREATE INDEX IX_FullTextSearch_Type "keyspace"."FullTextSearch" ("Type");
With the above table, I would need to insert rows for the text "Hello World" as follows:
BATCH APPLY;
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("H",000000000-0000-0000-0000-000000000,"Hello World",1);
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("He",000000000-0000-0000-0000-000000000,"Hello World",1);
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("Hel",000000000-0000-0000-0000-000000000,"Hello World",1);
.....
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("Hello Wor",000000000-0000-0000-0000-000000000,"Hello World",1);
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("Hello Worl",000000000-0000-0000-0000-000000000,"Hello World",1);
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("Hello World",000000000-0000-0000-0000-000000000,"Hello World",1);
.....
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("Wor",000000000-0000-0000-0000-000000000,"Hello World",1);
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("Worl",000000000-0000-0000-0000-000000000,"Hello World",1);
INSERT INTO "FullTextSearch" ("PartialText","TargetIdentifier","CompleteText","Type") VALUES ("World",000000000-0000-0000-0000-000000000,"Hello World",1);
END BATCH;
Basically, the above will satisfy the following wildcards/partialtext "%o W%", "Hello%", "Worl%"; However it will not satisfy partial words such as "%ell%" for "Hello", which I can feel alright about for now..... (OCD sorta kicks in here)
This approach sort of sucks for me because I would now have to delete/re-insert any time a save/name change occurs on the "TargetIdentifier";
The Second Solution, would be very similar only this time making use of wide-columns; where the table might look like:
CREATE TABLE "FullTextSearch" (
"TargetIdentifier" uuid,
"Type" int,
"CompleteText" text,
PRIMARY KEY("TargetIdentifier")
);
and now during a search something like:
SELECT * FROM "FullTextSearch" WHERE "He" = 1;
so that if the column exists, the respective rows are returned;
Third Solution:
similar to the one above, only this time instead of using wide-columns we use a set column such as map for the partial texts, and perform a query like:
SELECT * FROM "FullTextSearch" WHERE "PartialTexts"['He'] = 1;
Anyways, I am all out of ideas, it is late, and I can only hope for a great response! Please, let me know what I should be doing here... am I even on the right path?
AFAIK Datastax Enterprise Search is the (commercial) successor of Solandra.
Cassandra 2.0 supports so called "custom secondary indexes".
Custom secondary indexes are Java code. Your own implementation has to implement the abstract class org.apache.cassandra.db.index.SecondaryIndex
(See http://www.datastax.com/documentation/cql/3.1/cql/cql_reference/create_index_r.html)
I'm not sure whether implementations exist for Elasticsearch or Solr.
I would not recommend to code all the weird full text search logic like stemming, multiple/exotic language support or even geo spatial stuff.
But SecondaryIndexwould be a good point to start integrating your favorite search engine.
If your dataset is relative small you can simply use a inmemory instance of lucene, update the index at a set interval and you are ready to go.
Use elassandra which comes elasticsearch as a plugin in cassandra.
An example can be found from here
Check out SOLANDRA (former Lucandra)
But I think Solandra is not being actively developed any more, the author moved to Datastax and continued his work there.
So You can also take a look at Datastax Enterprise Search
There are some limitation also, look at DistributedSearch
The very basic thing about cassandra is if you want to use where clause for filtration of records that column is either primary key or you have to assign index to it, so what i can see is you have given primary key to "TargetIdentifier" field and index to "Type" and using "CompleteText" in where clause so this may not work..
Assign secondary index to "CompleteTex" and check whether you are getting desired output or not.
A couple other options you have:
Stratio Lucene Plugin. This uses Lucene for implementing a native secondary index.
You also have SSTable Attached Secondary Index (SASI) available to use for free text searching.
Be forewarned that both of these strategies use locally distributed indexes such that queries will not be very performant since searches will end up being broadcast across the entire cluster. For SASI, you can avoid this if you can use a partition key as part of your query.
Use Solr for fullText search
Cassandra is not good for fullText.
1 Db Cassandra for Archive
2 Solr for full text search
Can anyone provide an example of how to join across two schemas using subsonic 2.2.
This doesn't work:
SqlQuery qu = new Select("*")
.From(NorthwindLeads.Lead.Schema)
.InnerJoin(Northwind.StatsMap.SourceIdColumn, NorthwindLeads.Lead.SourceIdColumn);
There's no easy way to do this in subsonic that I'm aware of. I would recommend adding a view to your database which returns the data you want from the other schema and then joining to the view in your subsonic query.
I've got a Subsonic query that isn't returning any values. I think the problem is in my where clause, although I'm not sure why.
What I'm really looking for is a way to debug the query to see the SQL that's actually being spit out by Subsonic. I know there was a way to do this with the Query object with Inspect(), but I'm using Select objects (or could also probably use SQLQuerys) because I need joins. Is there any inspect() type option for a Subsonic Select?
Here's the code I'm using:
Dim qry As New [Select]("Contract_NO")
qry.From(<table1>.Schema)
qry.InnerJoin(<table2>.<table2columnA>, <table1>.<table1columnA)
qry.Where(NonInfoleaseLessor.Columns.LessorCode).Like("mystring")
If I comment out the where line, I get a full list of results. It doesn't like something about it, but I've manually run the query at the database with that where clause, and it works. How can I see what the difference is?
The problem with your query is that you should be using Contains("mystring") instead of Like("mystring").
The best way to see the SQL is to use the BuildSqlStatement() method of the query.
Use [a] profiler to see what SQL is actually being executed against the database.
As Adam spotted:
.Like("mystring")
should most probably be
.Like("%mystring%")
please try using Like("%mystring%")
It might have something to do with your choice of clause, or which column name you are using. Subsonic has a couple of column name field
OBJECT.xyzColumn
OBJECT.xyzColumn.QualifiedName
OBJECT.Columns.xyz
I have had to play with these in the past to get the values I wanted.
I want to no if there any way to add an "Having" constrain to an aggregation select?
Example: if i need all sales sum by date having the sales sum > 1000.
Best Regards,
TheGodfather
SubSonic does have a "having" but you don't explicitly state it.
It is determined from you selecting an Aggregate and adding the Aggregate to the Where clause.
For example (paraphrased from SubSonic AggregateTests.cs)
SubSonic.SqlQuery q = new
Select(Aggregate.GroupBy("ProductID"), Aggregate.Avg("UnitPrice"))
.From("Order Details")
.Where(Aggregate.Avg("UnitPrice"))
.IsGreaterThan(50);
The SubSonic query above will create a SQL statement with a "HAVING AVG(UnitPrice) > 50"
Are you using SubSonic 3.0.0.3 or 2.2?
If you're using 2.2, then I don't think you can do it. I'm unsure about 3.0.
I'd like to add a query to my SubSonic DAL that uses the MySQL fulltext search construct "WHERE MATCH (columnlist) AGAINST (searchterm)", but can't find an equivalent in SubSonic.
Is there a way of using Subsonic to execute a "literal" query - i.e. it just queries with the exact MySQl code I feed it?
Alternatively, could I implement a subclass of SubSonic.Where to run the fulltext search? If so, how would I go about this?
I'm using SubSonic 2.x. Any ideas welcome. Thanks.
ddoctor
You can use CodingHorror to run any SQL you like.