How could I query a levelDB database by values in nodejs?
I have key: value pairs in my leveldb database and I want to search some values in it and find that key-value pair from that result. I am a beginner in leveldb.
{
"key1" : {
"data1": "value1",
"data2": "value2",
"data3": "value3"
}
}
So, I want to query value where data3 == value3 and get that key-value pair.
In levelDB, we can only get data on the basis of key.
Is there any other alternative?
In leveldb, which a low level database abstraction you can "only" query by exact key match or prefix key range.
You can not query by value without somekind duplication.
What pattern I adopted in my graphdb project is to follow the EAV model with a secondary "table" to store the index.
In Python plyvel you can emulate "table" using prefixed databases. Or see how FoundationDB does it in its Subspace implementation. Basically, every key-value pair of a given "table" or "space" is prefixed with a particular bytes sequence, that is all.
The first table, looks like the following:
(Entity, Attribute) → (Value)
Where Entity is a (random) identifier and Attribute is the byte representation of field name and last but not least Value is the bytes serialized value associated with Attribute for the given Entity.
The table schema is done that way so that you can quickly fetch using a range query all Attribute and Value using prefix range search over a given Entity.
The index table use the following schema:
(Attribute, Value) → Entity
That is it a shuffled version of the first table.
This is done like so, to make it possible to quickly fetch Entity that match a particular Attribute == Value that's what you are looking for.
There is alternative implementations for what you are looking for. Lookup my answers about leveldb and key-value stores e.g. Expressing multiple columns in berkeley db in python?
Good luck!
Related
In hyperledger-fabric node js sdk.
Is there any possibility to search an asset with partial id?
for example my id is 'abc123'.
I can search with bc12 or abc or 123..and get the matching results.
Using stub.GetStateByRange(startKey, endKey) it is possible to retrieve results on a partial key, if they key has a specific form.
For eg.
the following keys could be used to successfully with a range query in the chaincode to retrieve a list of results, to match key abc123
a
ab
abc
abc1
abc12
abc123
However, a key without the same initial characters will not work. Eg. bc12 or 123.
The below function documentation gives a good idea of how the GetStateByRange function can be used.
// GetStateByRange returns a range iterator over a set of keys in the
// ledger. The iterator can be used to iterate over all keys
// between the startKey (inclusive) and endKey (exclusive).
// However, if the number of keys between startKey and endKey is greater than the
// totalQueryLimit (defined in core.yaml), this iterator cannot be used
// to fetch all keys (results will be capped by the totalQueryLimit).
// The keys are returned by the iterator in lexical order. Note
// that startKey and endKey can be empty string, which implies unbounded range
// query on start or end.
// Call Close() on the returned StateQueryIteratorInterface object when done.
// The query is re-executed during validation phase to ensure result set
// has not changed since transaction endorsement (phantom reads detected).
GetStateByRange(startKey, endKey string) (StateQueryIteratorInterface, error)
The answer by Clyde is the correct one to your question.
But, if you intend to perform complex queries in your code and you are in a position to refactor your data modelling, maybe you can set the information you must filter in some field inside your model (instead of or in addition to the ID itself) and perform rich queries against that field.
To do this, you must enable CouchDB as the state DB in your peers if haven't done it yet. Then you can query the DB and perform rich queries against your model fields.
Of course, this is not the answer to your question, but it may fit better to your use case if you are in a position to perform this kind of changes.
I have the following query:
SELECT *
FROM table
WHERE (id, other_id, status)
IN (
(1, 'XYZ', 'OK'),
(2, 'ZXY', 'OK') -- , ...
);
Is it possible to construct this query in a type-safe manner using jOOQ, preferably without generating composite keys? Is it possible to do this using jOOQ 3.11?
My apologies, it seems my Google-fu was not up to par. The opposite of this question can be found here: Use JOOQ to do a delete specifying multiple columns in a "not in" clause
For completeness' sake, so that other Google searches might be more immediately helpful, the solution is:
// can be populated using DSL.row(...); for each entry
Collection<? extends Row3<Long, String, String>> values = ...
dslContext.selectFrom(TABLE)
.where(DSL.row(ID, OTHER_ID, STATUS).in(values))
.fetch();
Relevant jOOQ documentation: https://www.jooq.org/doc/3.14/manual/sql-building/conditional-expressions/in-predicate-degree-n/
Your own answer already shows how to do this with a 1:1 translation from SQL to jOOQ using the IN predicate for degrees > 1.
Starting from jOOQ 3.14, there is also the option of using the new <embeddablePrimaryKeys/> flag in the code generator, which will produce embeddable types for all primary keys (and foreign keys referencing them). This will help never forget a key column on these queries, which is especially useful for joins.
Your query would look like this:
ctx.selectFrom(TABLE)
.where(TABLE.PK_NAME.in(
new PkNameRecord(1, "XYZ", "OK"),
new PkNameRecord(2, "ZXY", "OK")))
.fetch();
The query generated behind the scenes is the same as yours, using the 3 constraint columns for the predicate. If you add or remove a constraint from the key, the query will no longer compile. A join would look like this:
ctx.select()
.from(TABLE)
.join(OTHER_TABLE)
.on(TABLE.PK_NAME.eq(OTHER_TABLE.FK_NAME))
.fetch();
Or an implicit join would look like this:
ctx.select(OTHER_TABLE.table().fields(), OTHER_TABLE.fields())
.from(OTHER_TABLE)
.fetch();
consider this schema in dyanmo db,we count of question
[
{
'TableName': "user_detail",
'KeySchema': [
{'AttributeName': "timestamp", 'KeyType': "HASH"},
{'AttributeName': "question", 'KeyType': "RANGE"},
],
'AttributeDefinitions': [
{'AttributeName': "timestamp", 'AttributeType': "S"},
{'AttributeName': "question", 'AttributeType': "N"},
],
'ProvisionedThroughput': {
'ReadCapacityUnits': 40,
'WriteCapacityUnits': 40] }
}
]
I'm beginner of dyanmo db can any one give idea for that one.we need query,the sql query goes like that select count(question) from user_detail where question =1
Thanks in advance
I will throw some pointers. DynamoDB has two types of APIs :-
Option 1:-
1) Scan API - will scan the whole table. The scan api should be used when the hash key value is not known
2) Query API - will query the table using hash key. The hash key is must for Query API
In your case, the hash key value is not known. So, you can't use Query API. However, you can use scan API which is a very costly operation in terms of performance and cost. So, it should be avoided if you have a table of millions of items.
The alternative is to create global secondary index (GSI) with question attribute as hash key and some other field as sort key (possibly timestamp). This way, you should be able to use Query API on GSI. However, this wouldn't solve the problem completely.
DynamoDB doesn't have aggregate functions like count,min and max. So, you need to count the number of items in the result set at client side.
Option 2:-
If you have an option to change the data model, you can redesign the above table as mentioned below:-
question - hash key
timestamp - range key
I have seen many use cases using timestamp as range key. Please analyse your query access patterns (QAP) for all your use cases and make the decision accordingly.
Is there any solution to search by multiple string keys in the Couchbase?
For example I have an index:
emit(['example', doc.field1, doc.field2], doc.id);
And field1 and field2 are string variables.
I cannot use startkey and endkey because compound array index key is still treated as a string. So I have problem like this. But also I have string keys. So I cannot use Spatial Views.
Is there any solution other than a finding of multiple result sets for each part of the key, and then finding their intersection?
If you are limited to using views, then as you already discovered, there is no way to filter by independent fields.
If you are using (or can upgrade to) Couchbase 4.x, then you can do this with a simple N1QL query:
SELECT * FROM bucket WHERE field1 = 'bla' AND field2 ='blabla'
Be sure to create indexes on the two fields:
CREATE INDEX ix_field1 ON bucket(field1)
In Marklogic, I want to search between two collections by joining the id element of doc from collection1 to id element of doc from collection2. When it is matched i need the resulting document from both collections.
I have the below code, but it is very slow. How to use cts:search or search:search to achieve the same
for $i in collection('demographic')/individual,
$j in collection('membership')/membership[enrolleIndividualId/id/text() = $i/individual/id/text()])
return {$i,$j}
Update:
I should note that your sample is not valid XQuery: return element root { $i, $j } would be valid. Also, you should not use the /text() node selector, as it's behavior can be counterintuitive. You can compare elements directly in an XPath predicate ([enrolleIndividualId/id eq $i/individual/id]). Use /fn:string() in place of /text() if you need the contents of an element as a string. I'd also recommend using the atomic equality operator eq in place of the sequence equality operator = when directly comparing individual elements.
Original Answer:
There are several approaches to implementing joins in MarkLogic, but I would first question your data model. From the names of the elements in your sample query, it looks like you are using a relational model (individuals have memberships). MarkLogic is a document database, and it's optimized for denormalized documents. You will be much better served to process your data and generate new individual documents that each contain the relevant membership data.
That being said, here's how you could join your documents:
First, you will need range indices to write performant joins. If the id element from your sample query is not unique to individuals, you will need path range indices on enrolledIndividualId/id and individual/id, otherwise, a simple element range index on id will do.
The most common join pattern in MarkLogic uses a "shotgun-OR" query; first retrieving values from the lexicon backing a range index, and then constructing an or-query from those values to retrieve the relevant documents. This won't work directly in your case, as you want to retrieve both sides of the join. You can either run a search for each pair of documents, or run a single search for one side, and then an additional document read for each document.
pairs:
for $value in cts:values(cts:path-reference("individual/id"))
return
cts:search(/,
cts:or-query((
cts:and-query((
cts:collection-query("demographic"),
cts:path-range-query("individual/id", "=", $value))),
cts:and-query((
cts:collection-query("membership"),
cts:path-range-query("enrolledIndividualId/id", "=", $value))))),
"unfiltered")
shotgun-OR plus iteration:
for $doc in
cts:search(/,
cts:and-query((
cts:collection-query("demographic"),
cts:path-range-query("individual/id", "=",
cts:values(cts:path-reference("individual/id"))))),
"unfiltered")
return
cts:search(/,
cts:and-query((
cts:collection-query("membership"),
cts:path-range-query("enrolledIndividualId/id", "=", $doc/individual/id))),
"unfiltered")
As you can see, each approach requires I/O proportionate to the number of docs/values you want to join. If you only needed the shotgun-OR (ie, a query for documents based on criteria from other documents), you would only need to make two requests, the initial cts:values() call to retrieve values from a lexicon, and the cts:search() call using a query built from those values.
Note: the cts:query objects used in these examples could be used in conjunction with the Search API by means of the search:resolve() function.
Given your apparent data model, you will be much better served by processing your data into individual, de-normalized documents.