I'm working on a node.js project. I'm trying to understand how MongoDB works. I'm obtaining data hourly via a cron file. I'd like for there to be unique data, so I'm using update instead of insert. That works fine. I'd like to add the option that the data expires after three days. Its not clear to me how to do that.
In pseudo code:
Setup Vars, URL's, a couple of global variables, lineNr=1, end_index=# including databaseUrl.
MongoClient.connect(databaseUrl, function(err, db) {
assert.equal(null, err, "Database Connection Troubles: " + err);
**** db.collection('XYZ_Collection').createIndex({"createdAt": 1},
{expireAfterSeconds: 120}, function() {}); **** (update)
s = fs.createReadStream(text_file_directory + 'master_index.txt')
.pipe(es.split())
.pipe(es.mapSync(function(line) {
s.pause(); // pause the readstream
lineNr += 1;
getContentFunction(line, s);
if (lineNr > end_index) {
s.end();
}
})
.on('error', function() {
console.log('Error while reading file.');
})
.on('end', function() {
console.log('All done!');
})
);
function getContentFunction(line, stream){
(get content, format it, store it as flat JSON CleanedUpContent)
var go = InsertContentToDB(db, CleanedUpContent, function() {
stream.resume();
});
}
function InsertContentToDB(db, data, callback)
(expiration TTL code if placed here generates errors too..)
db.collection('XYZ_collection').update({
'ABC': data.abc,
'DEF': data.def)
}, {
"createdAt": new Date(),
'ABC': data.abc,
'DEF': data.def,
'Content': data.blah_blah
}, {
upsert: true
},
function(err, results) {
assert.equal(null, err, "MongoDB Troubles: " + err);
callback();
});
}
So the db.collection('').update() with two fields forms a compound index to ensure the data is unique. upsert = true allows for insertion or updates as appropriate. My data varies greatly. Some content is unique, other content is an update of prior submission. I think I have this unique insert or update function working correctly. Info from... and here
What I'd really like to add is an automatic expiration to the documents within the collection. I see lots of content, but I'm at a loss as to how to implement it.
If I try
db.collection('XYZ_collection')
.ensureIndex( { "createdAt": 1 },
{ expireAfterSeconds: 259200 } ); // three days
Error
/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/mongo_client.js:390
throw err
^
Error: Cannot use a writeConcern without a provided callback
at Db.ensureIndex (/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/db.js:1237:11)
at Collection.ensureIndex (/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/collection.js:1037:11)
at tempPrice (/var/lib/openshift/56d567467628e1717b000023/app-root/runtime/repo/get_options_prices.js:57:37)
at /opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/mongo_client.js:387:15
at process._tickCallback (node.js:442:13)
If I try to use createIndex I get this error...
`TypeError: Cannot call method 'createIndex' of undefined`
Note the database is totally empty, via db.XYZ_collection.drop() So yeah, I'm new to the Mongo stuff. Anybody understand what I need to do? One note, I'm very confused by something I read: in regards to you can't create TTL index if indexed field is already in use by another index. I think I'm okay, but its not clear to me.
There are some restrictions on choosing TTL Index: you can't create
TTL index if indexed field is already used in another index. index
can't have multiple fields. indexed field should be a Date bson type
As always, many thanks for your help.
Update: I've added the createIndex code above. With an empty callback, it runs without error, but the TTL system fails to remove entries at all, sigh.
Related
Recently, I changed key of object in MongoDB.
{
links: {
changed: 'value',
notchanged: 'value'
}
}
This is what I get from my MongoDB collection. Data which key is not changed is still readable by links.notchanged but data which key is changed like links.changed is not readable and only outputs undefined. Node.js gets and reads the whole links data correctly but when it comes to links.changed it doesn't. How do I solve this problem? Code below:
scheme.findOne({}, (err, data) => {
if (err) res.send('ERR')
else {
console.log(data) // prints full data, same as JSON above
console.log(data.links.changed) // undefined
}
}
You are matching {class:'210'}.. Is it available in document. Probably Your query returns empty object in data . Confirm the match query... Otherwise your code seems ok.
await db1.findOne({class: "210"}, (err, data) => {
console.log(data.links.changed) // returns value
})
Or Try the code like this
await db1.find({ class: "210" }).toArray()
.then(data => {
console.log(data[0].links.changed) //"value"
});
You should make a variable instead of an object for this. For example: Use changed and assign it value as true or false
Been trying to find samples usage for some of the static methods for a persistedModel in Loopback.
https://apidocs.strongloop.com/loopback/#persistedmodel-prototype-updateattribute
it just says:
persistedModel.updateAttributes(data, callback)
But how you I choose the which record I want to update? this is not working for me.
var order = Order.setId('whateverrecordId');
order.updateAttributes({name:'new name'},callback)
Loving loopback.. but their doc, sucks.. :(
You can use those on event listener like AfterSave
example:
Model.observe('after save', function(ctx, next) {
ctx.instance.updateAttribute(fieldname:'new value');
next();
});
1- What you did was right but i do not advise this method it's used for instance methods and generally to update fields like date for all the collection that you have so you don't need an id for it.
But you can try to make an array containing data to update containing also the ids and then make a comparison to fill in data for the ids that you have. (in #dosomething)
order.find().then(function(orders) {
orders.forEach(function(element) {
order.setId(element.id);
#DoSomething
order.updateAttribute({new: data}, function(err, instance) {
console.log(instance);
})
});
})
2- You can use updateAll to update one or many attribute.
PersistedModel.updateAll([where], data, callback)
var Updates = [{id : 1, name: name1}, ...]
Updates.forEach(function(element) {
order.updateAll({id : element.id}, {name :element.name}, function(err, count) {
if (err) {
console.error(err);
}
console.log(count); // number of data updated
})
})
I need some help to clear some things up.
I have a Model:
var Event = new Schema({
event_code: String
, segments: [Segment]
});
The creation of new documents work very well like perfect. When it comes to update certain documents I ran into some troubles.
When I do this (code below): = it only updates the first document, even if the id does not match
function edit_event (id, new_name, callback) {
Event.update(id, {$set:{event_code: new_name}}, function(err, doc) {
if (err) throw err;
callback();
});
}
When I do this (code below): = it gives me an Error (see below)
function edit_event (id, new_name, callback) {
Event.findByIdAndUpdate(id, {$set:{event_code: new_name}}, function(err, doc) {
if (err) throw err;
callback();
});
}
Error when using findByIdAndUpdate: Cast to ObjectId failed for value ""58fdbde31bff83141b376508"" at path "_id" for model "Event"
Please, i'm desperate :! :/
UPDATE
I figured out that the id that i'm trying to pass get stored with "" around it, so when i am looking for document with matching ID it puts an extra pair of "" around it so it ends up like ""id""
UPDATE 2
When I am listing all my documents, it returns:
{ _id: 58fdbde31bff83141b376508,
event_code: 'TestABC',
__v: 0,
segments: [] }
Then when i store the id in an HTML form it adds extra pair of "" around it ... that's not the case with event_code. Why is that ?
Looks like you performed unneeded JSON.stringify and that's why you get quotes around the id. Calling JSON.parse should solve the issue:
Event.findByIdAndUpdate(JSON.parse(id), {$set:{event_code: new_name}}, ...
I am parsing a CSV file, for each row I want to check if corresponding entry exists in the database, and if it does I want to update it, if it doesn't I want to enter a new entry.
It is very slow - only around 30 entries per second.
Am I doing something incorrectly?
Using node, mongodb, monk
function loadShopsCSV(ShopsName) {
var filename = 'test.csv'
csv
.fromPath(filename)
.on("data", function(data) {
var entry = {
PeriodEST: Date.parse(data[0]),
TextDate: textDateM,
ShopId: parseInt(data[1]),
ShopName: data[2],
State: data[3],
AreaUS: parseInt(data[4]),
AreaUSX: AreaUSArray[stateArray.indexOf(data[3])],
ProductClass: data[5],
Type: data[6],
SumNetVolume: parseInt(data[7]),
Weekday: weekdayNum,
WeightedAvgPrice: parseFloat(data[8]),
}
db.get(ShopsDBname).update(
{"PeriodEST" : entry.PeriodEST,
"ShopName": entry.ShopName,
"State" : entry.State,
"AreaUS" : entry.AreaUS,
"ProductClass" : entry.ProductClass,
"Type" : entry.Type},
{$set : entry},
function(err, result) {
}
);
}
}
})
.on("end", function() {
console.log('finished loading: '+ShopsName)
});
}, function(err) {
console.error(err);
});
}
First I would suggest to localize problem:
replace .on("data", function(data) with dummy .on("data", function() {return;}) and confirm speed of csv parsing.
turn on mongo profiler db.setProfilingLevel(1) and check slow log if there is any query slower than 100 ms.
If there are no problems above - the bottleneck is in one of nodejs libraries you are using to prepare and send query.
Assuming the problem is with slow mongodb queries, you can use explain for the update query for details. It may be the case it does not use any indexes and run a table scan for every update.
Finally, it is recommended to use bulk operations, which was designed for exactly your usecase.
Have you tried updating with no write concern? as MongoDB blocks until whole update is successful and DB sends back that acknowledgement? Are you on cluster or something? (might want to write into primary node if so)
after your {$set : entry},
{writeConcern: {w: 0}}
This is likely a simple answer but I'm relatively new to asynchronous programming and I'm looking for somebody to point me in the right direction.
My question is this - What is the best way to go about finding or creating a number of documents from an array of names (I'm using Mongoose) and then returning an array of _id's?
So to be clear, I want to:
Given an array of names, find or create a document with each name
Return an array of the existing or newly created documents _ids
You can use async module and within it it's async.parallel() method -
https://github.com/caolan/async#quick-examples
async.parallel([
function(){ ... },
function(){ ... }
], callback);
Or you can use promises and then Q.all() to get the array of ids back -
https://github.com/kriskowal/q#combination
Q.all(arrayOfFindOps).then(function(rows) {
return _.pluck(rows, '_id')
}
If you don't want to use any of the above and do it with callbacks, then you have to keep track of the count of array length, keep adding the ids to an array and when your completion counter reaches array length, call another function with the array you made.
This code can be easily modified to meet your requirements. Call this function for each document that required to be created if doesn't exist.
function(req, reply) {
// return document. if not found, create docuemnt.
docModel.findOne( {'name': req.params.name}, function ( err , doc) {
if(err){
//handle error
}
if(doc===null){
//find failed, time to create.
doc = new docModel( {'name': req.params.name} );
doc.save(function(err){
if(err){
//handle error
}
});
}
return reply(user._id);
});
}