I am making an application in nodejs in which i have to create cassandra database through CQL.
I have tried to use helenus,cassandra-driver but didn't able to create keyspace through CQL.
I have also tried apollo-cassandra and also able to connect to DB and create keyspace and enter tuples but when we fire queries again after restarting the application then it is showing schime mis-match error.
Which nodejs driver for cassandra is simple and fully supported for CQL?
Datastax has an official driver in the companys github repo.
https://github.com/datastax/nodejs-driver
The driver is for Apache Cassandra. This driver works exclusively with the Cassandra Query Language version 3 (CQL3) and Cassandra's native protocol.
High level features:
Automatic node discovery
Simple, prepared and batch statement support
Load balancing policies
Failover with retry and reconnection policies
SASL auth support
It works really well and is actively mantained.
Another route (if you prefer not writing the query statement outright in your code) is CassanKnex, it uses the datastax driver to execute queries - usage:
var cassanKnex = require("cassanknex")({
connection: { // default is 'undefined'
contactPoints: ["10.0.0.2"]
},
exec: { // default is '{}'
prepare: false // default is 'true'
}
});
cassanKnex.on("ready", function (err) {
var qb.createKeyspace("cassanKnexy")
.withSimpleStrategy(1);
.exec(function(err, res) {
if (err)
console.error("error", err);
else
console.log("res", res);
});
}
});
Related
I am using spark streaming for reading TCP server and then inserting the data into Cassandra, which I have to further push to UI, for pushing I decided to go for NodeJS. But I am not getting any technology which can talk with Cassandra as well as NodeJS. Below is my architecture, and I am not able to find out technology which can replace ? I am also open to change Cassandra with MongoDB if it is possible to directly push data from Mongo to NodeJS. But as of now I am using Cassandra as it have native support for Hadoop.
Take a look at the datastax nodejs-cassandra driver: https://github.com/datastax/nodejs-driver project. This has cassandra row streaming and piping functionality that you could use to push cassandra data into node, process, and then export via websockets per your desired architecture.
leave your stream client open - and this would need to run as a persistent node server handling errors - something like this should pick up new cassandra data:
var streamCassandra = function(){
client.stream('SELECT time, val FROM temperature WHERE station_id=', ['abc'])
.on('readable', function () {
//readable is emitted as soon a row is received and parsed
var row;
while (row = this.read()) {
console.log('time %s and value %s', row.time, row.val);
}})
.on('error', function (err) {
//handle err
})
.on('end', streamCassandra);
};
wrap your stream client into a recursive function that calls itself again on('end', streamCassandra). you could also poll the function every x seconds with a setInterval if you dont need that kind of concurrency. One of those approaches should work
Have you checked NiFi?
https://nifi.apache.org/
In your case, you could write your Spark Streaming results to Kafka, HDFS, or even directly to NiFi, but I personally prefer to write to Kafka or some other message queue.
From NiFi, you can write to Kafka, and also send requests to your Node JS app if that's what you need. In my case, I'm using Meteor, so just pushing from Kafka to MongoDB automatically refreshes the UI.
I hope it helps.
My current scenario is like
I have a rabbit mq which gives me the details of the order placed.
On the other side I have my titan db (cassandra storage, es index backends and gremlin server).
Yet another I have nodejs application which can interact with gremlin server through http api using https://www.npmjs.com/package/gremlin . I am able to make hits to my graph database from here.
Now what I am trying to do is load data from rabbit mq into titan db.
What I have been able to do till now is load the data from nodejs file using gremlin node module
var createClient = require('gremlin').createClient;
//import { createClient } from 'gremlin';
const client = createClient();
client.execute('tx=graph.newTransaction();tx.addVertex(T.label,"product","id",991);tx.commit()', {}, function(err, results){
if (err) {
return console.error(err)
}
console.log(results)
});
How should I move next so that I can harness existing rabbit mq of orders and push them into titan db.
Due to some constraints I can not use java.
You're most likely looking for something like node-amqp, which is a Node.js client for RabbitMQ. What you want to do is:
Establish a connection to Gremlin Server
Establish a connection to RabbitMQ
Listen to a RabbitMQ queue for messages
Send these messages to Gremlin, creating graph elements
Things you must watch for that will otherwise likely kill your performance:
Send Gremlin queries with bound parameters
Batch messages: create multiple vertices and commit them in the same transaction (= same Gremlin query, unless in session mode where you .commit() yourself). Numbers in the couple thousands should work.
Watchout for back-pressure and make sure you don't flood your Titan instances with more messages than they can handle.
I'm not familiar with RabbitMQ but hopefully this should get you started.
Note: Gremlin javascript driver interacts with Gremlin Server via a WebSocket connection, which is permanent and bi-directional. The client doesn't support the HTTP Channelizer yet (which is not the kind of connection that you wish to establish in the current scenario).
I have few integration tests that use real MongoDB database version 3.2. Tests are run with Mocha and I'm dropping the test db after each test with following code. It uses Mongoose version 4.4.12 and native MongoDB Node.js driver.
afterEach((done) => {
const connection = mongoose.createConnection(config.db.host);
connection.once("open", () => {
connection.db.command({ dropDatabase: 1 })
.then((res) => done(null, res))
.catch(done);
});
})
It worked nicely until I enabled sharding in my local environment to cover that with tests as well. Dropping database started to fail randomly. Above code fails to drop my test db about on every third test run. My sharding setup is minimal and have one config server, one mongos and two mongo shards.
After I added some logging I noticed that connection.db.command({ dropDatabase: 1 }) returns { info: 'database does not exist', ok: 1 } on those failed runs. Database is not dropped because MongoDB thinks it is already dropped even it is not the case.
Am I doing it completely wrong? Is there some Mongoose / MondoDB Node.js native driver configuration or such that I'm missing here? Or is this some known problem? Any help is much appreciated.
EDIT: Same thing happens if I use MongoClient directly, so Mongoose is not causing this.
We are using a node stack to connect to a mongo replica set. Because our replicas are geographically distributed, we would like to use the readPreference option in the URI and set it to nearest. But when we do so, while performance is greatly improved, we start getting "not master" errors.
Am I misunderstanding the use of the flag?
We are using mongo 2.6.3 and we are using version 2.0.24 of the mongodb node library.
The URI for the connection is:
mongodb://mongo-1:27017,mongo-2:27017,mongo-3:27017,mongo-4:27017,mongo-5:27017/db?replicaSet=main&readPreference=nearest
Burc
Option 1:
You could append slaveOk to end of connection URI. readPreference tells mongodb that how you'd like to read data and slaveOk instructs that it's OK to read from secondaries (bit duplicate) but works.
e.g.
mongodb://mongo-1:27017,mongo-2:27017,mongo-3:27017,mongo-4:27017,mongo-5:27017/db?replicaSet=main&readPreference=nearest&slaveOk=true
please see &slaveOk=true and end of URI.
https://mongodb.github.io/node-mongodb-native/driver-articles/mongoclient.html#read-preference
Option 2:
if above solution is not working, you'll need to modify code:
var client = require('mongodb').MongoClient;
var uri= "mongodb://mongo-1:27017,mongo-2:27017,mongo-3:27017,mongo-4:27017,mongo-5:27017/db?replicaSet=main";
Please note that I have modified connection uri. Instead of setting readPrefference in Uri, I moved it to as db option in MongoClient.connect.
var options = {db:{readPreference:"ReadPreference.NEAREST"}};
client.connect(uri, options, function(err, db){
if(err){
console.log(err);
return;
}
db = db.collection('data');
db.findOne({}, function(err, result){
console.log(result);
});
});
I have tested in nodejs driver 2.2 and hopping it should work in 2.0 version too.
It seems like there was a bug in the driver which was fixed in 2.0.28, where findAndModify used the readPreference setting. Upgrading the driver to the latest release seemed to have fixed the problem.
2.0.x drivers history
i am using node-oracleDB connector to connect to my local database , and i want to ask if it was an other(optimal) way to connect to the database, in order to improve communication ability between nodejs and oracle. thanks
I think using connection pooling is the good way. You can find some examples of pooling at here.
Please look at this file https://github.com/oracle/node-oracledb/blob/master/test/pool.js and read carefully to understand how handling connections in a pool.
Node-oracledb is the connector to use when writing Node.js applications that need to connect to Oracle Database.
Most apps will want to use a connection pool
var oracledb = require('oracledb');
oracledb.createPool (
{
user : "hr"
password : "welcome"
connectString : "localhost/XE"
},
function(err, pool)
{
pool.getConnection (
function(err, connection)
{
// use the connection
connection.execute(. . .
// release the connection when finished
connection.close(
function(err)
{
if (err) { console.error(err.message); }
});
});
});
});
There are some simplifications such as the 'default' connection pool that make it easier to share a pool across different modules. This is all covered in the documentation.