I have a table created as follows:
CREATE TABLE my_table (
date text,
id text,
time timestamp,
value text,
PRIMARY KEY (id));
CREATE INDEX recordings_date_ci ON recordings (date);
I'm able to simply add a new row to the table using the following Node code:
const cassandra = require('cassandra-driver');
const client = new cassandra.Client({ contactPoints: ['localhost'], keyspace: 'my_keyspace'});
const query = 'INSERT INTO my_table (date, id, time, url) VALUES (?, ?, ?, ?)';
client.execute(query, ['20160901', '0000000000', '2016-09-01 00:00:00+0000', 'random url'], function(err, result) {
if (err){
console.log(err);
}
console.log('Insert row ended:' + result);
});
However, I get the following error:
'Error: Expected 8 or 0 byte long for date (24)
When I change the timestamp to epoc time:
client.execute(query, ['20160901', '0000000000', 1472688000, 'random url']
I get:
d
OverflowError: normalized days too large to fit in a C int
I'm able to insert new rows via cqlsh so I'm probably missing something with the node.js driver.
Any idea?
Thanks
Where you have a string 2016-09-01 00:00:00+0000, instead use new Date('2016-09-01 00:00:00+0000').
Related
I am working on a project that will upload some records to SQL Server from a node.js program. Right now, this is my approach (inside an async function):
con = await sql.connect(`mssql://${SQL.user}:${SQL.password}#${SQL.server}/${SQL.database}?encrypt=true`);
for (r of RECORDS) {
columns = `([column1], [column2], [column3])`;
values = `(#col1, #col2, #col3)`;
await con
.request()
.input("col1", sql.Int, r.col1)
.input("col2", sql.VarChar, r.col2)
.input("col3", sql.VarChar, r.col3)
.query(`INSERT INTO [dbo].[table1] ${columns} VALUES ${values}`);
}
Where records is an array of objects in the form:
RECORDS = [
{ col1: 1, col2: "asd", col3: "A" },
{ col1: 2, col2: "qwerty", col3: "B" },
// ...
];
This code works, nevertheless, I have the feeling that it is not efficient at all. I have an upload of around 4k records and it takes roughly 10 minutes, it does not look good.
I believe if I can create a single query - instead of wrapping single inserts inside a for loop - with all the record values it will be faster, and I know there is a syntax for reaching that in SQL:
INSERT INTO table1 (column1, column2, column3) VALUES (1, "asd", "A"), (2, "qwerty", "B"), (...);
However I cannot find any documentation from mssql module for node on how to prepare the parameterized inputs to do everything in a single transaction.
Can anyone guide me into the right direction?
Thanks in advance.
Also, very similar to the bulk insert, you can use a table valued parameter.
sql.connect("mssql://${SQL.user}:${SQL.password}#${SQL.server}/${SQL.database}?encrypt=true")
.then(() => {
const table = new sql.Table();
table.columns.add('col1', sql.Int);
table.columns.add('col2', sql.VarChar(20));
table.columns.add('col3', sql.VarChar(20));
// add data
table.rows.add(1, 'asd', 'A');
table.rows.add(2, 'qwerty', 'B');
const request = new sql.Request();
request.input('table1', table);
request.execute('procMyProcedure', function (err, recordsets, returnValue) {
console.dir(JSON.stringify(recordsets[0][0]));
res.end(JSON.stringify(recordsets[0][0]));
});
});
And then for the SQL side, create a user defined table type
CREATE TYPE typeMyType AS TABLE
(
Col1 int,
Col2 varchar(20),
Col3 varchar(20)
)
And then use this in the stored procedure
CREATE PROCEDURE procMyProcedure
#table1 typeMyType READONLY
AS
BEGIN
INSERT INTO table1 (Col1, Col2, Col3)
SELECT Col1, Col2, Col3
FROM #MyRecords
END
This gives you more control over the data and lets you do more with the data in sql before you actually insert.
As pointed out by #JoaquinAlvarez, bulk insert should be used as replied here: Bulk inserting with Node mssql package
For my case, the code was like:
return await sql.connect(`mssql://${SQL.user}:${SQL.password}#${SQL.server}/${SQL.database}?encrypt=true`).then(() => {
table = new sql.Table("table1");
table.create = true;
table.columns.add("column1", sql.Int, { nullable: false });
table.columns.add("column2", sql.VarChar, { length: Infinity, nullable: true });
table.columns.add("column3", sql.VarChar(250), { nullable: true });
// add here rows to insert into the table
for (r of RECORDS) {
table.rows.add(r.col1, r.col2, r.col3);
}
return new sql.Request().bulk(table);
});
The SQL data types have to match (obviously) the column type of the existing table table1. Note the case of column2, which is a column defined in SQL as varchar(max).
Thanks Joaquin! I went down on the time significantly from 10 minutes to a few seconds
I am trying to join two tables: users and favourites. There is a possibility that a user has no favourites yet and when I tried to INNER JOIN the two I didn't get back the user without any favourites. Is there any way to join even if the second table has no data for that user?
I created the users tabel with the following code:
db.run(`CREATE TABLE Users(
UserId INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
Name TEXT NOT NULL,
Password TEXT NOT NULL,
Phone VARCHAR,
Email TEXT,
RestaurantId INTEGER,
FOREIGN KEY(RestaurantId) REFERENCES Restaurants(RestaurantId))`, (err) => {
if(err) {
console.error(err.message);
} else {
//insert some values
var insert = 'INSERT INTO Users (Name, Password, Phone, Email, RestaurantId) VALUES (?, ?, ?, ?, ?)';
db.run(insert, [
'Liam',
'blabla',
'+32412345678',
'email#email.com',
1
]);
}
}
);
And the favourites table with:
db.run(`CREATE TABLE Favourites(
FavouriteId INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
UserId INTEGER NOT NULL,
RestaurantId INTEGER NOT NULL,
FOREIGN KEY(UserId) REFERENCES Users(UserId),
FOREIGN KEY(RestaurantId) REFERENCES Restaurants(RestaurantId))`, (err) => {
if(err) {
console.error(err.message);
} else {
//insert some values
var insert = 'INSERT INTO Favourites (UserId, RestaurantId) VALUES (?, ?)';
db.run(insert, [
1,
1
]);
}
}
);
There is no problem with the data that exists in the table that was generated after these inserts. The problem only exists when a new user without favourites gets added to the database.
You are looking for LEFT JOIN. Take a look at the documentation: https://www.w3resource.com/sqlite/sqlite-left-join.php.
LEFT JOIN returns all the records on the left side of the join, with the matched records from the right side.
I am trying to get last inserted row in cassandra but I am getting undefined
Here is how code looks for insert:
const cassandra = require('cassandra-driver');
const client = new cassandra.Client({ contactPoints: ['h1', 'h2'], keyspace: 'ks1' });
let query = "INSERT INTO users (id, name, email ) values (uuid(), ?, ?)";
client.execute(query, [ 'badis', 'badis#badis.com' ])
.then(result => console.log('User with id %s', result.rows[0].id));
Remember you are dealing with NoSQL ("non relational")
If I were to run a SELECT * FROM users LIMIT 1 on this table, my
result set would contain a single row. That row would be the one
containing the lowest hashed value of username (my partition key),
because that's how Cassandra stores data in the cluster. I would have
no way of knowing if it was the last one added or not, so this
wouldn't be terribly useful to you.
The workings are very different from those that you might be used to in MySQL.
If obtaining the last inserted row's information is needed for your system, then you will have to know it before you insert it.
Generating UUID
const Uuid = require('cassandra-driver').types.Uuid;
const id = Uuid.random();
Then you can use the id as another value for the prepared insert query
const cassandra = require('cassandra-driver');
const client = new cassandra.Client({ contactPoints: ['h1', 'h2'], keyspace: 'ks1' });
let query = "INSERT INTO users (id, name, email ) values (?, ?, ?)";
client.execute(query, [id, 'badis', 'badis#badis.com' ])
.then(result => console.log('User with id %s', id));
I am trying cassandra node driver and stuck in problem while inserting a record, it looks like cassandra driver is not able to insert float values.
Problem: When passing int value for insertion in db, api gives following error:
Debug: hapi, internal, implementation, error
ResponseError: Expected 4 or 0 byte int (8)
at FrameReader.readError (/home/gaurav/Gaurav-Drive/code/nodejsWorkspace/cassandraTest/node_modules/cassandra-driver/lib/readers.js:291:13)
at Parser.parseError (/home/gaurav/Gaurav-Drive/code/nodejsWorkspace/cassandraTest/node_modules/cassandra-driver/lib/streams.js:185:45)
at Parser.parseBody (/home/gaurav/Gaurav-Drive/code/nodejsWorkspace/cassandraTest/node_modules/cassandra-driver/lib/streams.js:167:19)
at Parser._transform (/home/gaurav/Gaurav-Drive/code/nodejsWorkspace/cassandraTest/node_modules/cassandra-driver/lib/streams.js:101:10)
at Parser.Transform._read (_stream_transform.js:179:10)
at Parser.Transform._write (_stream_transform.js:167:12)
at doWrite (_stream_writable.js:225:10)
at writeOrBuffer (_stream_writable.js:215:5)
at Parser.Writable.write (_stream_writable.js:182:11)
at write (_stream_readable.js:601:24)
I am trying to execute following query from code:
INSERT INTO ragchews.user
(uid ,iid ,jid ,jpass ,rateCount ,numOfratedUser ,hndl ,interests ,locX ,locY ,city )
VALUES
('uid_1',{'iid1'},'jid_1','pass_1',25, 10, {'NEX1231'}, {'MUSIC'}, 21.321, 43.235, 'delhi');
parameter passed to execute() is
var params = [uid, iid, jid, jpass, rateCount, numOfratedUser, hndl, interest, locx, locy, city];
where
var locx = 32.09;
var locy = 54.90;
and call to execute looks like:
var addUserQuery = 'INSERT INTO ragchews.user (uid ,iid ,jid ,jpass ,rateCount ,numOfratedUser ,hndl ,interests ,locX ,locY ,city) VALUES (?,?,?,?,?,?,?,?,?,?,?);';
var addUser = function(user, cb){
console.log(user);
client.execute(addUserQuery, user, function(err, result){
if(err){
throw err;
}
cb(result);
});
};
CREATE TABLE ragchews.user(
uid varchar,
iid set<varchar>,
jid varchar,
jpass varchar,
rateCount int,
numOfratedUser int,
hndl set<varchar>,
interests set<varchar>,
locX float,
locY float,
city varchar,
favorite map<varchar, varchar>,
PRIMARY KEY(uid)
);
P.S
Some observations while trying to understand the issue:
Since it seems, problem is with float so i changed type float (of locX, locY) to int and re-run the code. Same error persist. Hence, it is not problem associated specifically to float CQL type.
Next, i attempted to remove all int from the INSERT query and attempted to insert only non-numeric values. This attempt successfully inputted the value into db. Hence it looks like now that, this problem may be associated with numeric types.
Following words are as it is picked from cassandra node driver data type documentation
When encoding data, on a normal execute with parameters, the driver tries to guess the target type based on the input type. Values of type Number will be encoded as double (as Number is double / IEEE 754 value).
Consider the following example:
var key = 1000;
client.execute('SELECT * FROM table1 where key = ?', [key], callback);
If the key column is of type int, the execution fails. There are two possible ways to avoid this type of problem:
Prepare the data (recommended) - prepare the query before execution
client.execute('SELECT * FROM table1 where key = ?', [key], { prepare : true }, callback);
Hinting the target types - Hint: the first parameter is an integer`
client.execute('SELECT * FROM table1 where key = ?', [key], { hints : ['int'] }, callback);
If you are dealing with batch update then this issue may be of your interest.
I am using helenus in my node-js project to get/set values in cassandra. I have a MapType field inside my Table, but when I retrieve the value from the table, I get an empty key-value set.
Below is the schema for my table
CREATE TABLE datapoints (
id uuid PRIMARY KEY,
created_at timestamp,
properties map<text,text>
);
I have inserted the values from cql using the query below
INSERT INTO datapoints (id, properties) VALUES (24053e20-63e9-11e3-8d81-0002a5d5c51b, { 'fruit' : 'apple', 'band' : 'Beatles' });
Below is my nodejs code:
var helenus = require('/usr/local/lib/node_modules/helenus')
var pool = new helenus.ConnectionPool({
hosts : ['localhost:9160'],
keyspace : 'mykeyspace',
timeout : 3000
});
pool.connect(function(err, keyspace){
if(err){
console.log("connect me error")
throw(err);
} else {
pool.cql("SELECT * FROM datapoints", [], function(err,results){
console.log("results", results)
results.forEach(function(row){
props = row.get('properties').value;
var id = row.get('id').value;
console.log("properties", props);
console.log("id", id);
});
})
}
});
The line console.log("properties", props); returns me a function, and when I call that function, I get an empty key value set. Please help.
It seems there was an issue with the deserialization of the collection types. The pull request that was made in the past broke the deserialization. I just pushed a fix to version 0.6.8 that should take care of this.