I am using a dynamodb database and attempting to update a value via node.js. I have created a table called counter in order to determine the current count of students stored in a separate table. I have had luck updating string values, but have had nothing but trouble trying to update this counter value. I have tried many different configurations of code over the last several hours to no avail. Any help would be appreciated.
function updateCount(operand) {
var params = {
TableName : "counter",
Key:{
"type" : {
"S" : "student"
}
},
UpdateExpression : "SET pos = pos + :o",
ExpressionAttributeValues:{
":o": Number(operand)
},
ReturnValues:"UPDATED_NEW"
}
docClient.update(params, function(err, data) {
if (err)
console.log(err);
else {
console.log(data);
res.send(JSON.stringify(data, null, 2));
}
});
}
updateCount(1);
The current error shows this but I've had many more:
C:\Users\Ko_Kor\Desktop\CS 496\assignment#3>node server.js
Express started on http://localhost:8081; press Ctrl-C to terminate.
{ [ValidationException: The provided key element does not match the schema]
message: 'The provided key element does not match the schema',
code: 'ValidationException',
time: Fri Oct 21 2016 20:36:43 GMT-0700 (Pacific Daylight Time),
requestId: 'T8G9V8KF587THVK5OTGQ95TH9VVV4KQNSO5AEMVJF66Q9ASUAAJG',
statusCode: 400,
retryable: false,
retryDelay: 0 }
I have also tried using this format:
ExpressionAttributeValues:{
":o": {"N": operand}
},
DynamoDB doesn't provide this kind of feature.
You can try to play with DynamoDB atomic counters, but I will suggest to try another approach like string UUID.
You could use ADD
[...]
UpdateExpression: "ADD counter :o SET updatedAt = :updatedAt",
ExpressionAttributeValues: {
':o': valueToIncrementWith,
':updatedAt': Date.now()
}
[...]
Related
I'm trying to batch update table users that contains these columns id (primary key) , status (text) , active (numeric).
the array i'm receiving from back-end is something like this:
[
{ id: 33715, status: 'online', active: 10 },
{ id: 39129, status: 'offline', active: 0.1 },
{ id: 36090, status: 'loggedin', active: 24 },
{ id: 34452, status: 'loggedout', active: 1 },
]
active is time in hours. now i want to bulk update this array into users table. as each object represents a row in a table.
I've tried this approach according to this solution Patrick Motard
function bulkUpdate (records) {
var updateQuery = [
'INSERT INTO users (id, status, active) VALUES',
_.map(records, () => '(?)').join(','),
'ON DUPLICATE KEY UPDATE',
'status = VALUES(status),',
'active = VALUES(active)'
].join(' '),
vals = [];
_(records).map(record => {
vals.push(_(record).values());
});
return knex.raw(updateQuery, vals)
.catch(err => {
console.log(err)
});
}
bulkUpdate(response);
but i get this error
error: syntax error at or near "DUPLICATE"
so what i'm missing here. and does anyone by chance have a better solution without using promises or bluebird then do trx.commit , this consumes large cpu and ram. and doesn't do the purpose of update 10,000 row at once
I don't see any ON DUPLICATE KEY UPDATE reference in
https://www.postgresql.org/docs/9.5/sql-insert.html
You could try out ON CONFLICT DO UPDATE SET. Try out first with plain SQL to write a query that seems to work correctly and then it is easy to convert to javascript generated one.
I am trying to build a "number of visitors" collection in mongoDb using Node.JS backend of my website. The frontend sends the following info to Node.JS backend as JSON.
isUniqueVisitor - 1 if yes, 0 if no
country - standard country code - "JP", "IN", "UK", etc
My database looks like following
{
"today": 2019-06-07,
"uniqueVisitors": {
"count": 230,
"countries": {
"JP": 102,
"IN": 88,
"UK": 30
}
}
}
It works well if I use $inc with fixed values
Eg. $inc: {count: 1} // for string/integers keys
Eg. $inc: {"uniqueVisitors.count": 1} // inside quotes to access key of a JSON
Main issue:
I am not able to access a document name using variable.
Eg. $inc: {`uniqueVisitors.countries[${req.body.country}]`}
This creates an error as backticks can't be used for Mongo.
I tried with
Eg. $inc: {uniqueVisitors["countries"][req.body.country]}
But even this creates error.
I followed the web and found that mongo $set using variables can be realized by passing the required JSON directly to $set. Hence I resorted to code it the following way.
mongoClient.connect(mongoURL, async function (err, db) {
if (err) throw err;
console.log("Database connected");
// Identifying my document with today's date
var myQuery = {
date: getTodayDate()
};
// Defining the JSON to be passed to uniqueVisitors $inc
var uniqueVisitorsInc = {
"uniqueVisitors": {
"count": 0,
"countries": {}
}
};
// Populating the JSON to be passed to uniqueVisitors $inc => essentially asking to increase count by 1 and increase that country's count by 1
uniqueVisitorsInc["uniqueVisitors"]["count"] = 1;
uniqueVisitorsInc["uniqueVisitors"]["countries"][myData.country] = 1;
var newValues = {
$inc: uniqueVisitorsInc
};
await db.collection("visitorStats").update(myQuery, newValues, {upsert: true});
db.close();
});
The above method worked well on editor but threw the following runtime error:
$inc requires numerical values
Basically asking me to pass values to $inc in {var1: 1, var2: 5} pattern.
Please help me bypass this weird situation.
I know I can do a two step process where I read the values first, increment in variable and $set it in Mongo.
But does anyone know how to overcome this situation using $inc?
If this update were hardcoded to update "JP" only, it'd need to look like:
$inc: { "uniqueVisitors.country.JP": 1 }
So you were almost there with the backtick method but change the syntax a bit and keep the : 1 part like so:
$inc: { [`uniqueVisitors.country.${req.body.country}`]: 1 }
I'm beginner of dynamoDB
I want to know how solve this problem.
I want get MapInfo.MapX value(I want to get low level attribute of MapInfo)
const params = {
TableName: "ZzimCuration",
ProjectionExpression: "Id, Knock, Modified, MapInfo.MapX"
}
dynamo.doc.scan(params, (err, data) => {
if(err){
console.error("No Scan : " ,JSON.stringify(err, null, 2))
} else{
console.log(data);
but results like this
{
Knock: 3811,
Id: 'tour.2029011',
Modified: 20161201153517,
MapInfo: [Object]
}
I don't want rebuild json.
How can I get MapInfo.mapX directly???
By default, console log display only the first level property of
objects
Change the console log in the else statement as mentioned below:-
console.log(JSON.stringify(data, null, 2));
I want to create a scheduled job for patients in a hospital. The patients will be informed every month by their reg_date.
I'm using new Date().getDate() inside my scheduled Jobs to run at 8.00 AM in the morning to send SMS to my patients. Meanwhile, I had been using string format date to save reg_date in my mongoDB. Here is snippets of my mongoDB docs :
{
customer: "John",
reg_date: "2017-02-17T16:39:26.969Z"
}
I've ben surfing for solutions but it turns out nothing, so I decided to post myself. Here is what i am trying to do :
customer.find({"reg_date.getDate()" : new Date(2017, 03, 17).getDate()})
.then(function(data) {
for (var key in data.length) {
sendTheSMS(key[data]);
};
});
E.g: What I am doing is "I want to get every patient who register at 17th day of the month and send them a SMS".
Any help will be appreciated. :D
For this type of bit complex query you need to use aggregation method instead regular find method.
$project this will help you to project your fields, here we are creating a new temporary field day with only date of the reg_date. Then we query using the new field day and we get the result.
This temp field day will never added to your schema or model, it is just like temp view we are creating like in SQL.
Here i projected only customer and day but Please project all the fields necessary in the result.
function getCustomerList(day, callback){
customer.aggregate([
{
$project:{
"customer": "$customer", //repeat the same for all field you want in result
"reg_date": "$reg_date",
"day":{$dayOfMonth:"$reg_date"} //put day of month in 'day'
}
},
{
$match:{
"day": day //now match the day with the incoming day value
}
},
], function(err, result){
callback(err, result);
})
}
getCustomerList(17, function(err, result){ // call the function like this with date you want
// Process the err & result here
});
Result will be like this
[{
"_id" : ObjectId("571f2da8ca97eb10163e6e17"),
"customer" : "John",
"reg_date" : ISODate("2016-04-17T08:58:16.414Z"),
"day" : 17
},
{
"_id" : ObjectId("571f2da8ca97eb10163e6e17"),
"customer" : "Prasanth",
"reg_date" : ISODate("2016-04-17T08:58:16.414Z"),
"day" : 17
}]
Ignore the day field projected during your process...
With reg_date in string you can't query for day of month as it only works with ISODate. I suggest first you convert the string in reg_date in all your documents with a script.
Then the following query should work
customer.aggregate([
{
$project:{
"document": "$$ROOT", //to get the whole document
"day":{$dayOfMonth:"$date"} //put day of month in 'day'
}
},
{
$match:{
"day": 17 //match 17
}
},
], function(data) {
for (var key in data.length) {
sendTheSMS(key[data]);
};
})
Use greater than and less than
var previousDate =new Date(2017, 1, 16); //month starts with 0
var nextDate=new Date(2017, 1, 18);
customer.find({reg_date : { $gt:previousDate,$lt:nextDate}})
.then(function(data) {
for (var key in data.length) {
sendTheSMS(key[data]);
};
});
Since reg_date is stored as a string, and not a Date/ISODate, you're limited as to what kind of query you can run (so I concur with the comment in one of the other answers that you should consider converting them to proper ISODate).
Considering that you want to query a date string for entries with a particular day-of-month, you can use a regular expression query:
customer.find({ reg_date : /-17T/ })
Or, dynamically:
let today = new Date();
let dom = ('00' + today.getDate()).slice(-2); // zero-pad day of month
let re = new RegExp('-' + dom + 'T');
customer.find({ reg_date : re })
You should also read this regarding speed optimizations, but still, regex queries aren't very fast.
I'm working on a node.js project. I'm trying to understand how MongoDB works. I'm obtaining data hourly via a cron file. I'd like for there to be unique data, so I'm using update instead of insert. That works fine. I'd like to add the option that the data expires after three days. Its not clear to me how to do that.
In pseudo code:
Setup Vars, URL's, a couple of global variables, lineNr=1, end_index=# including databaseUrl.
MongoClient.connect(databaseUrl, function(err, db) {
assert.equal(null, err, "Database Connection Troubles: " + err);
**** db.collection('XYZ_Collection').createIndex({"createdAt": 1},
{expireAfterSeconds: 120}, function() {}); **** (update)
s = fs.createReadStream(text_file_directory + 'master_index.txt')
.pipe(es.split())
.pipe(es.mapSync(function(line) {
s.pause(); // pause the readstream
lineNr += 1;
getContentFunction(line, s);
if (lineNr > end_index) {
s.end();
}
})
.on('error', function() {
console.log('Error while reading file.');
})
.on('end', function() {
console.log('All done!');
})
);
function getContentFunction(line, stream){
(get content, format it, store it as flat JSON CleanedUpContent)
var go = InsertContentToDB(db, CleanedUpContent, function() {
stream.resume();
});
}
function InsertContentToDB(db, data, callback)
(expiration TTL code if placed here generates errors too..)
db.collection('XYZ_collection').update({
'ABC': data.abc,
'DEF': data.def)
}, {
"createdAt": new Date(),
'ABC': data.abc,
'DEF': data.def,
'Content': data.blah_blah
}, {
upsert: true
},
function(err, results) {
assert.equal(null, err, "MongoDB Troubles: " + err);
callback();
});
}
So the db.collection('').update() with two fields forms a compound index to ensure the data is unique. upsert = true allows for insertion or updates as appropriate. My data varies greatly. Some content is unique, other content is an update of prior submission. I think I have this unique insert or update function working correctly. Info from... and here
What I'd really like to add is an automatic expiration to the documents within the collection. I see lots of content, but I'm at a loss as to how to implement it.
If I try
db.collection('XYZ_collection')
.ensureIndex( { "createdAt": 1 },
{ expireAfterSeconds: 259200 } ); // three days
Error
/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/mongo_client.js:390
throw err
^
Error: Cannot use a writeConcern without a provided callback
at Db.ensureIndex (/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/db.js:1237:11)
at Collection.ensureIndex (/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/collection.js:1037:11)
at tempPrice (/var/lib/openshift/56d567467628e1717b000023/app-root/runtime/repo/get_options_prices.js:57:37)
at /opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/mongo_client.js:387:15
at process._tickCallback (node.js:442:13)
If I try to use createIndex I get this error...
`TypeError: Cannot call method 'createIndex' of undefined`
Note the database is totally empty, via db.XYZ_collection.drop() So yeah, I'm new to the Mongo stuff. Anybody understand what I need to do? One note, I'm very confused by something I read: in regards to you can't create TTL index if indexed field is already in use by another index. I think I'm okay, but its not clear to me.
There are some restrictions on choosing TTL Index: you can't create
TTL index if indexed field is already used in another index. index
can't have multiple fields. indexed field should be a Date bson type
As always, many thanks for your help.
Update: I've added the createIndex code above. With an empty callback, it runs without error, but the TTL system fails to remove entries at all, sigh.