While using Meteor, I sometimes access the underlying Node Mongo driver so I can make bulk updates and inserts.
const bulk = Coll.rawCollection().initializeOrderedBulkOp();
bulk.insert({key_id: Mongo.Collection.ObjectID()}); // note key_id is an ObjectID
...
bulk.execute();
But the value of the key_id fields ends up being the plain subdocument {_str: '...'} when I look in the database after the insert.
Is there any way to use bulk operations in Node's Mongo library (whatever it is Meteor uses) and keep ObjectID's as Mongo's ObjectID type?
(There's many posts about the nature of the different ID types, and explaining Minimongo, etc. I'm interested specifically about the bulk operations converting ObjectID's into plain objects, and solving that issue.)
From Neil's top-level comment
On a native method you would actually need to grab the native implementation. You should be able to access from the loaded driver through MongoInternals [...]
Mongo.Collection.ObjectID is not a plain ObjectId representation, and is actually a complex object for Meteor internal use. Hence why the native methods don't know how to use the value.
So if you have some field which is an ObjectId, and you're using some method of a Meteor Collection's rawCollection (for example,
.distinct
.aggregate
.initializeOrderedBulkOp
.initializeUnorderedBulkOp
), you'll want to convert your ObjectId's using
const convertedID = new MongoInternals.NpmModule.ObjectID(
originalID._str
);
// then use in one of the arguments to your function or something
const query = {_id: convertedID};
before calling the method on them.
Related
I am trying to query an old mongoDB collection in node.js with the native driver.
It uses Legacy UUID as its _id field.
In my code I am using Binary values such as: "2u0kLwUZuEWQvjqOjgQU4g=="
and I want to query the collection with find() operation.
When trying to test it directly with mongoDB I am able to use the following query:
db.getCollection('myCollection').find({_id: BinData(3, "2u0kLwUZuEWQvjqOjgQU4g==")})
to find my item.
But what I need to do is to find multiple items.
So I need to use something like this:
db.getCollection('myCollection').find({_ids: { $in: [BinData(3, "2u0kLwUZuEWQvjqOjgQU4g=="), BinData(3, "3u0kLwUZuEWQvjqOjgQU4g==")...]}})
but it does not seem to work and always returns zero records.
I am not sure why? and what might be the correct way to query multiple Legacy UUIDs?
Recently I have decided to replace arrays with priority queues for storing my list of jobs for a user into MongoDB. I use NodeJS and ExpressJS for backend. The priority queue I attempted to store is from an external package which can be installed by running the following command in terminal:
yarn add js-priority-queue
For some reason the priority queue works perfectly prior to storing it into MongoDB. However, the next time I attempt to take it out of MongoDB and use it, its functionality is missing. I declare its type as Schema.Types.Mixed in the Schema. Am I doing something wrong or is it not possible to store instantiated class objects into MongoDB?
As far as I know, when you store things in MongoDB they are stored as extended JSON (EJSON) in binary format (BSON)
const { EJSON } = require('bson');
const test = EJSON.stringify({a: new Date(), foo:function(){console.log('foo');}})
console.log(test) // "{"a":{"$date":"2020-07-07T14:45:49.475Z"}}"
So any sort of function is lost.
I recently started using Node.js + Express.js (generated with pug) + pg-promise for handling db.
My first target is to obtain data from Postgres (already set up) and display it pretty using render and pug. Let's say it is user list from Users table.
On this restful tutorial I have learned how to get data and return it as JSON - it worked.
Based on Mozilla's tutorial I seperated my code:
routes/users.js: where for '/' I call user_controller.user_list method (using router.get)
controllers/userController.js I have exported user_list where I would like to ask model for data and call render if I have results
queries.js which is kinda my model? But I'm not sure. It has API: connection to db with promises and one function for every query I am going to use in Controllers. I believe I should have like one Model file per table (or any logical entity) but where to store pgp connections?
This file is based on first tutorial I mentioned
// queries.js (connectionString is set properly to my postgres)
var pgp = require('pg-promise')(options);
var db = pgp(connectionString);
function getUsers(req, res, next) {
db.any('SELECT (user_id, username) FROM public.users ORDER BY user_id ASC LIMIT 1000')
.then(function (data) {
res.json({ data: data });
})
.catch(function (err) {
return next(err);
});
}
module.exports = {
getUsers: getUsers
};
Here starts my problem as most tutorials uses mongoose which is very model-db-schema-friendly and what I have is simple 'SELECT ...' string I pass to pg-promise's any() function.
Therefore I have no model class like User.
In userControllers.js I don't know how to call getUsers() to handle its data. Returning JS object from getUsers() would be nice.
Also: where should I call render? In controller or only in
db.any(...).then(function (data) { <--here--> })
Before, I also tried to embed whole Postgres handling into Controller but from db.any() I got this array for handling:
[{ row: '(1,John)' },{ row: '(2,Amy)' },{ row: '(50,Peter)' } ]
Didn't know how go from there as I probably lost my API functionality as well ;-)
I am browsing through multiple tutorials how to handle MVC but usually they handle MongoDB and
satisfy readers with res.send() not render().
I am not sure that I understand what your question is exactly about, but since I do not have enough reputation to comment, I'll do my best to help you with your interrogations. :)
First, regarding the queries.js file, it is IMO not exactly a model, but rather a DAO (Data Access Object) file. DAO comes between you Model (which is actually you database) and your Controller layers. There usually is a DAO file per object (User, Pet, whatever you want) in your data model.
When the data model is rather complex, it can be useful to use an Object Relational Mapping (ORM) such as Mongoose to map your database and execute complexe processes on your objects. In such a case, you might need a specific file per object so as to describe your model and store your queries. But since you don't need an ORM, you DAO can directly interact with your database. That is why you do not have a User.js file.
Regarding the way the db object should be used, I think you should refer directly to pg-promise documentation on the matter.
IMPORTANT: For any given connection, you should only create a single
Database object in a separate module, to be shared in your application
(see the code example below). If instead you keep creating the
Database object dynamically, your application will suffer from loss in
performance, and will be getting a warning in a development
environment (when NODE_ENV = development)
As a matter of fact, a db object in pg-promise sort of represents the database itself and is actually designed for the simultaneous use of several databases, which does not seem to be your case for the moment.
Finally, when it comes to the render function, I believe it should be in the controller, as your DAO is not supposed to know how the data it has gathered is going to be used.
Modularity is always a time-saving choice on the long-term.
Furthermore, note that you might later need a Business Layer between your DAO and your controller, in order to preprocess and postprocess data you are going to persist or to display. In such a case, if you need for instance to ask for data from your database, you will need to render data after it is processed by the Business layer. If the render is made in the DAO layer, it will not be possible.
In the link I provided earlier to pg-promise's db object connection, you will also find documentation on the any() method. You might already have looked it up.
It specifically states that it returns
A promise object that represents the query result:
When no rows are returned, it resolves with an empty array.
When 1 or more rows are returned, it resolves with the array of rows.
so your returned data is a JS Array. If you want to make it a JS object, just use
JSON.stringify(yourArray) to process your data before rendering it in your controller.
But I wonder if Pug is not able to use your data directly.
Also, if you cannot get any data out of your DAO, maybe you should check that your data object is not empty, as such a case is tolerated by the any() method. If you expect your query to always return something, you might want to consider using the many() or the one() methods.
I hope this helps you.
I have a question regarding compound index creation in mongodb:
Say I want to create this compound index:
cSchema.index({account:1, authorization: 1, c_type: 1});
Problem is, javascript doesn't guarantee dictionary order, so I won't be sure the compound index is in the order I want.
How can I make sure it's indeed {account:1, authorization:1, c_type:1} in that order ?
Thanks !
The simple answer is that most abuse the behaviour that simple String properties that don't parse to an integer on an Object will enumerate in creation order. Although not guaranteed in ES2015 for some enumeration methods, it does work in a confined environment like Node/V8. This has worked in most JS engines for a while now but had never been part of the ES spec before ES2015.
MongoDB
The underlying MongoDB driver createIndex function supports String, Array and Object index definitions. The parsing code is in a util function called parseIndexOptions.
If you specify an array of Strings, Array pairs, or Objects then the order will be fixed:
['location','type']
[['location', '2d'],['type', 1]]
[{location: '2d'},{type: 1}]
Also note that the createIndex code does use Object.keys to enumerate when it gets an Object of index properties.
Mongoose
The Schema#index function is documented as requiring an Object to pass to "MongoDB driver's createIndex() function" so looks to support passing through whatever options you provide.
There are a couple of places where indexes are further processed though. For example when you create a sub document with an index, the index needs to have the parent schema prefixed onto field names. On a quick glance I think this code still works with an Array but I can't see any tests for that in the Mongoose code so you might want to confirm it yourself.
Mongoose does have a compound index test that relies on Object property order.
Is there a way to tell the native MongoDB driver for NodeJS to automatically convert the contents of an _id field into an ObjectID?
Say, in this situation:
db.collection("collection").updateOne({_id: data._id}, data)
It's not that data._id = ObjectID(data.id) is hard, but it's another thing to miss each and every time.
There is no way to do that natively. You can make some function for wrapping your mongo queries where you will check params and if it's "_id" parse it to ObjectId.