I'm working with the Datastax Node.js driver and I can't figure out when to use a mapper vs. query. Both seem to be able to perform the same CRUD operations.
With a query:
const q = SELECT * FROM mykeyspace.mytable WHERE id='12345';
client.execute(q).then(result => console.log('This is the data', result);
With mapper:
const tableRow = await tableMapper.find({ id: '12345' });
When should I use the mapper over a query and vice versa?
Mapper is a feature from cassandra-driver released in 2018. Using mapper, cassandra-driver can make a map from your cassandra table to an object in nodejs and you can handle in your nodejs application like a set of document.
Using mapper you can make selects or inserts in your database like said in this article:
https://www.datastax.com/blog/2018/12/introducing-datastax-nodejs-mapper-apache-cassandra
With query method, if you need to use or reuse any property from your json you will need to make a Json.Parse().
The short answer is: whatever you find more comfortable.
The Mapper lets you deal with database data as documents (JavaScript objects), builds the CQL query for you, executes the query and maps the results.
On the other hand, the core driver only supports executing CQL queries that you have to write yourself.
Related
I recently started using Node.js + Express.js (generated with pug) + pg-promise for handling db.
My first target is to obtain data from Postgres (already set up) and display it pretty using render and pug. Let's say it is user list from Users table.
On this restful tutorial I have learned how to get data and return it as JSON - it worked.
Based on Mozilla's tutorial I seperated my code:
routes/users.js: where for '/' I call user_controller.user_list method (using router.get)
controllers/userController.js I have exported user_list where I would like to ask model for data and call render if I have results
queries.js which is kinda my model? But I'm not sure. It has API: connection to db with promises and one function for every query I am going to use in Controllers. I believe I should have like one Model file per table (or any logical entity) but where to store pgp connections?
This file is based on first tutorial I mentioned
// queries.js (connectionString is set properly to my postgres)
var pgp = require('pg-promise')(options);
var db = pgp(connectionString);
function getUsers(req, res, next) {
db.any('SELECT (user_id, username) FROM public.users ORDER BY user_id ASC LIMIT 1000')
.then(function (data) {
res.json({ data: data });
})
.catch(function (err) {
return next(err);
});
}
module.exports = {
getUsers: getUsers
};
Here starts my problem as most tutorials uses mongoose which is very model-db-schema-friendly and what I have is simple 'SELECT ...' string I pass to pg-promise's any() function.
Therefore I have no model class like User.
In userControllers.js I don't know how to call getUsers() to handle its data. Returning JS object from getUsers() would be nice.
Also: where should I call render? In controller or only in
db.any(...).then(function (data) { <--here--> })
Before, I also tried to embed whole Postgres handling into Controller but from db.any() I got this array for handling:
[{ row: '(1,John)' },{ row: '(2,Amy)' },{ row: '(50,Peter)' } ]
Didn't know how go from there as I probably lost my API functionality as well ;-)
I am browsing through multiple tutorials how to handle MVC but usually they handle MongoDB and
satisfy readers with res.send() not render().
I am not sure that I understand what your question is exactly about, but since I do not have enough reputation to comment, I'll do my best to help you with your interrogations. :)
First, regarding the queries.js file, it is IMO not exactly a model, but rather a DAO (Data Access Object) file. DAO comes between you Model (which is actually you database) and your Controller layers. There usually is a DAO file per object (User, Pet, whatever you want) in your data model.
When the data model is rather complex, it can be useful to use an Object Relational Mapping (ORM) such as Mongoose to map your database and execute complexe processes on your objects. In such a case, you might need a specific file per object so as to describe your model and store your queries. But since you don't need an ORM, you DAO can directly interact with your database. That is why you do not have a User.js file.
Regarding the way the db object should be used, I think you should refer directly to pg-promise documentation on the matter.
IMPORTANT: For any given connection, you should only create a single
Database object in a separate module, to be shared in your application
(see the code example below). If instead you keep creating the
Database object dynamically, your application will suffer from loss in
performance, and will be getting a warning in a development
environment (when NODE_ENV = development)
As a matter of fact, a db object in pg-promise sort of represents the database itself and is actually designed for the simultaneous use of several databases, which does not seem to be your case for the moment.
Finally, when it comes to the render function, I believe it should be in the controller, as your DAO is not supposed to know how the data it has gathered is going to be used.
Modularity is always a time-saving choice on the long-term.
Furthermore, note that you might later need a Business Layer between your DAO and your controller, in order to preprocess and postprocess data you are going to persist or to display. In such a case, if you need for instance to ask for data from your database, you will need to render data after it is processed by the Business layer. If the render is made in the DAO layer, it will not be possible.
In the link I provided earlier to pg-promise's db object connection, you will also find documentation on the any() method. You might already have looked it up.
It specifically states that it returns
A promise object that represents the query result:
When no rows are returned, it resolves with an empty array.
When 1 or more rows are returned, it resolves with the array of rows.
so your returned data is a JS Array. If you want to make it a JS object, just use
JSON.stringify(yourArray) to process your data before rendering it in your controller.
But I wonder if Pug is not able to use your data directly.
Also, if you cannot get any data out of your DAO, maybe you should check that your data object is not empty, as such a case is tolerated by the any() method. If you expect your query to always return something, you might want to consider using the many() or the one() methods.
I hope this helps you.
While using Meteor, I sometimes access the underlying Node Mongo driver so I can make bulk updates and inserts.
const bulk = Coll.rawCollection().initializeOrderedBulkOp();
bulk.insert({key_id: Mongo.Collection.ObjectID()}); // note key_id is an ObjectID
...
bulk.execute();
But the value of the key_id fields ends up being the plain subdocument {_str: '...'} when I look in the database after the insert.
Is there any way to use bulk operations in Node's Mongo library (whatever it is Meteor uses) and keep ObjectID's as Mongo's ObjectID type?
(There's many posts about the nature of the different ID types, and explaining Minimongo, etc. I'm interested specifically about the bulk operations converting ObjectID's into plain objects, and solving that issue.)
From Neil's top-level comment
On a native method you would actually need to grab the native implementation. You should be able to access from the loaded driver through MongoInternals [...]
Mongo.Collection.ObjectID is not a plain ObjectId representation, and is actually a complex object for Meteor internal use. Hence why the native methods don't know how to use the value.
So if you have some field which is an ObjectId, and you're using some method of a Meteor Collection's rawCollection (for example,
.distinct
.aggregate
.initializeOrderedBulkOp
.initializeUnorderedBulkOp
), you'll want to convert your ObjectId's using
const convertedID = new MongoInternals.NpmModule.ObjectID(
originalID._str
);
// then use in one of the arguments to your function or something
const query = {_id: convertedID};
before calling the method on them.
I am building a REST API which connects to a NEO4J instance. I am using the koa-neo4j library as the basis (https://github.com/assister-ai/koa-neo4j-starter-kit). I am a beginner at all these technologies but thanks to some help from this forum I have the basic functionality working. For example the below code allows me to create a new node with the label "metric" and set the name and dateAdded propertis.
URL:
/metric?metricName=Test&dateAdded=2/21/2017
index.js
app.defineAPI({
method: 'POST',
route: '/api/v1/imm/metric',
cypherQueryFile: './src/api/v1/imm/metric/createMetric.cyp'
});
createMetric.cyp"
CREATE (n:metric {
name: $metricName,
dateAdded: $dateAdded
})
return ID(n) as id
However, I am struggling to know how I can approach more complicated examples. How can I handle situations when I don't know how many properties will be added when creating a new node beforehand or when I want to create multiple nodes in a single post statement. Ideally I would like to be able to pass something like JSON as part of the POST which would contain all of the nodes, labels and properties that I want to create. Is something like this possible? I tried using the below Cypher query and passing a JSON string in the POST body but it didn't work.
UNWIND $props AS properties
CREATE (n:metric)
SET n = properties
RETURN n
Would I be better off switching tothe Neo4j Rest API instead of the BOLT protocol and the KOA-NEO4J framework. From my research I thought it was better to use BOLT but I want to have a Rest API as the middle layer between my front and back end so I am willing to change over if this will be easier in the longer term.
Thanks for the help!
Your Cypher syntax is bad in a couple of ways.
UNWIND only accepts a collection as its argument, not a string.
SET n = properties is only legal if properties is a map, not a string.
This query should work for creating a single node (assuming that $props is a map containing all the properties you want to store with the newly created node):
CREATE (n:metric $props)
RETURN n
If you want to create multiple nodes, then this query (essentially the same as yours) should work (but only if $prop_collection is a collection of maps):
UNWIND $prop_collection AS props
CREATE (n:metric)
SET n = props
RETURN n
I too have faced difficulties when trying to pass complex types as arguments to neo4j, this has to do with type conversions between js and cypher over bolt and there is not much one could do except for filing an issue in the official neo4j JavaScript driver repo. koa-neo4j uses the official driver under the hood.
One way to go about such scenarios in koa-neo4j is using JavaScript to manipulate the arguments before sending to Cypher:
https://github.com/assister-ai/koa-neo4j#preprocess-lifecycle
Also possible to further manipulate the results of a Cypher query using postProcess lifecycle hook:
https://github.com/assister-ai/koa-neo4j#postprocess-lifecycle
The DataStax documentation says that to page through all data, the following CQL query is useful:
SELECT * FROM test WHERE token(k) > token(42);
Is it possible to build this query using the QueryBuilder? It provides a token method, but that seems to work only on column names, not on values.
Ideally, the value (in the example: 42) is of type Object, just like in the eq/gte/lte functions.
Try using automatic paging with the .fetchSize method. It uses token under the hood:
Automatic paging is introduced Cassandra 2.0. Automatic paging allows the developer to iterate on an entire ResultSet without having to care about its size: some extra rows are fetched as the client code iterate over the results while the old ones are dropped. The amount of rows that must be retrieved can be parameterized at query time. In the Java Driver this will looks like:
Statement stmt = new SimpleStatement("SELECT * FROM images");
stmt.setFetchSize(100);
ResultSet rs = session.execute(stmt);
Source: http://www.datastax.com/dev/blog/client-side-improvements-in-cassandra-2-0
QueryBuilder.fcall("token", value) ;
can solve the problem！
Maybe the question does not apply to dynamoDB due to it not being Relational Db.
However, I'm looking for a good object mapper which can be used in nodejs and aws sdk to map existing model classes to dynamoDB tables. Does anyone have experience with this issue/question, or have you used such a module/library?
If you are looking for schema:
https://github.com/clarkie/dynogels (well supported forked from vogels which has been abandoned)
https://github.com/automategreen/dynamoose (inspired by Mongoose)
If you are looking for something to throw javascript objects (even circular graphs) to:
https://github.com/aaaristo/dyngodb (alpha)
https://github.com/aaaristo/angular-gson-express-dyngodb
dyngodb has experimental support for full-text search, and transactions too.
Both are based on aws-sdk.
Also worth considering is simple marshallers, which just translate between the dynamoDB format and regular js objects or JSON.
DynamoDb-Data-Types
https://github.com/kayomarz/dynamodb-data-types
https://www.npmjs.com/package/dynamodb-data-types
"This utility helps represent AWS DynamoDb data types. It maps (marshalls) JavaScript data into the format required by DynamoDb."
dynamoDb-marshaler
https://github.com/CascadeEnergy/dynamoDb-marshaler
https://www.npmjs.com/package/dynamodb-marshaler
"Translates sane javascript objects (and JSON) into DynamoDb format and vice versa." [does not support B type.]
Update 2016-06:
Just discovered that the AWS SDK now does this for you. Their documentation is only partially converted so I guess this is a recent addition. Read about it here.
But these marshallers are still useful because there are circumstances where you can't use the new document client, eg. when processing a dynamoDB stream.
You could also try: https://dynamoosejs.com/. It is inspired by mongoose again.
If you are using Typescript, dynamo-easy might be a good option. Just add some decorators to your model and start using it.
import { Model, PartitionKey, DynamoStore } from '#shiftcoders/dynamo-easy'
#Model()
export class Person {
#PartitionKey()
id: string
name: string
yearOfBirth: number
}
const personStore = new DynamoStore(Person)
personStore
.scan()
.whereAttribute('yearOfBirth').equals(1958)
.exec()
.then(res => console.log('ALL items with yearOfBirth == 1958', res))
It uses the AWS DynamoDB sdk but takes care of the mapping between JS and DynamoDB types and provides a simple to use fluent API.
full disclosure: I am one of the authors of this library
After looking over all the posts I landed on https://github.com/awspilot/dynamodb-oop
It doesn't hide the API but instead just wraps it in a nice, fluent way with promises even and you inject your version of the aws-sdk. It's similar to dynamodb-data-types but also wraps the methods too (not just the data types).
Extra bonus, the same author has https://github.com/awspilot/dynamodb-sql Didn't use the sql wrapper but I can see how some people may prefer that.
Dynamoose is obviously inspired by mongoose and is a good choice if you have a well-defined schema and/or want to be abstracted away from the DynamoDB details.
Have you seen dynasaur? It seems to be the type of thing you're looking for, but I have not used it myself. There's also dynamodb-data-types which is not an ORM, but makes it easy to convert to/from standard JavaScript objects.