MongoDB's BSON "_id" looks like scientific notation in Node.js - node.js

Occasionally I get a document with an _id value which javascript could and does interpret as scientific notation.
ONLY illustrate the problem I ran the following query.
db.users.find({$where:'this._id > 1'}).count()
2
There are hundreds of docs in this collection, but those 2 that evaluate as numbers cause problems when they get used in {$in:[]} clauses.
db.users.findOne({$where:'this._id > 1'})._id
ObjectId("5225141850742e0000002331") - see it's looks like scientific notation right?
I think I run into trouble when I want to store that _id as a string in another collection say as
friendsCollection:
{
_uid:"5225141850742e0000002331"
//more properties here
}
When I retrieve that value, Node (or Mongoose) interprets it as a number like "Infinity". Then my code ends up trying to search for the user with {_id:{$in:[Infinity]}} which throws an error.
I'm guessing there's a more robust way to store _id values or to handle properties you know to be _ids, but I don't know how.

Converting from hex string to binary representation of ObjectID
If you want to convert from a 24-byte hex string representation of an _id value into a binary ObjectID as stored in MongoDB, you can use ObjectID.createFromHexString:
// Need to require ObjectID class if not already included
ObjectID = require('mongodb').ObjectID;
var uid = ObjectID.createFromHexString("5205c4bd7c21105d0d99648c")
Comparing ObjectIDs
You should be using the $gt operator for ObjectID comparison rather than $where. The $where operator evaluates a JavaScript string and cannot take advantage of indexes; it will be much less performant (particularly for this use case).
So the findOne() example to find an _id greater than a given ObjectID should instead be:
db.users.findOne(
{ _id: { $gt: ObjectID("5205c4bd7c21105d0d99648c") } }
)._id
For a predictable outcome on finding the next higher ObjectID you should specify an explicit sort order using find() with sort and limit:
// Find next _id greater than ObjectID("5205c4bd7c21105d0d99648c")
db.users.find(
{_id: { $gt: ObjectID("5205c4bd7c21105d0d99648c") } }
).sort({_id:1}).limit(1).toArray()[0]._id
You'll notice that these find examples doesn't explicitly call createFromHexString. The default ObjectID() constructor will try to create an appropriate ObjectID based on whether the given value is a 24 byte hex string, 12 byte binary string, or a Number. If you know what sort of value you are providing, it is better to call the expected constructor to limit unexpected conversions (for example if you accidentally provided a Number instead of a hex string).
Database References (DBRefs)
MongoDB explicitly does not support joins, however there is a convention for storing database references (DBRefs) when you want to store the _id of a related document as a reference in another document. Mongoose has a ref option that simplifies working with references; see 'Population' in the Mongoose docs.

At some point I had problems when querying _id using the native driver. I fixed it by using ObjectId in my queries. You might find this helpful:
var ObjectId = require("mongodb").ObjectID;
query._id = { $gt: ObjectId(idString) }

Ahhh, maybe I should use the Mongoose data type "ObjectId" in my schemas.
For example I was using mongoose schemas like this:
locations:{
_uid:{type:String},//<--probably not good
lat:{type:Number},
lng:{type:Number}
},
Started using the Schema type "ObjectId":
locations:{
_uid:Schema.Types.ObjectId,//<--better?
lat: {type:Number},
lng: {type:Number}
},
I have to rebuild a lot of data to support this change. Also, I won't be sure until another one of those scientific notation _ids pop up, (I deleted the offending docs). However, this seams promising.

Related

What is MongoDB aggregate pipeline?

Using Mongoose driver
Consider the following code :
Connecting to database
const mongoose = require("mongoose");
require("dotenv").config();
mongoose.connect(process.env.DB);
const userSchema = new mongoose.Schema({ name: String }, {collection: 'test'});
const Model = mongoose.model('test', userSchema);
Creating dummy document
async function createDocs() {
await Model.deleteMany({});
await Model.insertMany([{name: "User1"}, {name: "User2"}, {name: "User3"},{name: "User4"}])
}
createDocs();
Filtering data using Model.find()
async function findDoc () {
let doc = await Model.find({name: 'User1'});
console.log(`Using find method : ${doc}`);
}
findDoc();
Filtering data using Model.aggregate()
async function matchDoc() {
let doc = await Model.aggregate([
{
$match: {name : 'User1'}
}
])
console.log(`Using aggregate pipeline : `, doc);
}
matchDoc();
• Both the processes produce the same output
Q1) What is an aggregate pipeline and why use it?
Q2) Which method of retrieving data is faster?
I will not get too much into this as there is a lot of information online. But essentially the aggregation pipeline gives you access to a lot of strong operators - mainly used for data analysis and object restructuring, for simple get and set operations there is no use for it.
A "real life" example of when you'd want to use the aggregation pipeline is if you want to calculate an avg of a certain value in your data, obviously this is just the tip of the iceberg in terms of power that this feature allows.
find is slightly faster, the aggregation pipeline has some overhead when compared to the query language as each stage has to load the BSON documents into memory, where find doesn't. If your use case is indeed just a simple query then find is the way to go.
You are checking a smaller piece of the aggregation pipeline.
You cannot do pipeline with a single query using find
Let's say you want to find all the orders which have a product that was purchased within a duration. Here, orders and customers are two different collections, You need multiple stages.
Let's say you stored data in a different format, For ex, date as a string, integers as a decimal. If you want to convert on the fly, you can use aggregation.
If you want to use aggregation operators in an update operation from mongo 4.2+, It helps.
You can restructure data in find. Let's say I just need a few fields after aggregation from an array field.
You cannot find a particular array or object element matching a condition using simple find. elemMatch is not that powerful.
You cannot group things with simple find
And many more.
I request you to check aggregate operators and relevant examples from the documentation
Data retrieving depends on not depend on the aggregation or find. It depends on the data, hardware, and index settings.

Change string indexes to be ObjectID ones in large MongoDB instance

So, I've git a large production database dump with _id field as strings. Different collections got those string's length different. There're a lot of relations there. I need a way to change string _ids to ObjectId ones.
What I've tried already:
1) Looking mongoose/mongodb documentation for single command to do that failed
2) node.js migration script that grabs all the entries in one collection and wraps string id into ObjectId just fails either because of stack overflow FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory if we're trying to delete-and-recreate approach or with an error about bad string lenght and impossibility to create ObjectId from that string.
Will attach data samples and/or mongoose schema a bit later.
A simple and inefficient cursor solution to avoid JavaScript heap out of memory is to serialize everytyhing. Of course in order to edit the _id you have to create a new document and remove the old one.
const cursor = Model.find().lean().cursor();
let doc;
while ((doc = await cursor.next())) {
// The string must be a valid ObjectId, otherwhise it won't work
if (typeof doc._id === 'string') {
let newId = new mongoose.Types.ObjectId(doc._id);
let newDoc = new Model(Object.assign({}, doc, {_id: newId}));
await newDoc.save();
await Model.remove({_id: doc._id});
}
}
However, if you have errors about incorrect ids, it may be beacause the string id are not actually stringified version of mongo ObjectId. In such case, the relation cannot be preserved.

How to Join two collections in MongoDB and NodeJS with $lookup and DbRef?

If i have two collections where one of that have dbref, how is possibile to join using $lookup and dbref?
DBref is a BSON Object and you cannot make a lookup using its value.
BUT, there is a way to do it, transforming a DBRef object into an array.
I have written an answer a few months ago describing how to do it.
Short explanation
Say you have DBRef object like this :
myField: DBRef("otherCollection", ObjectId("582abcd85d2dfa67f44127e0")),
Use $objectToArray on myField like this
db.myColl.aggregate([
{
$project: {
transformedDBRef: {$objectToArray: "$myField"},
}
},
])
The result would be an array of two objects, one object for the reference, one for the ObjectId contained within the DBRef, each with a field "k" and a field "v". It will look like this:
transformedDBRef: [{"k" : "$ref","v" : "otherCollection"},{"k" : "$id","v" : ObjectId("582abcd85d2dfa67f44127e0")}
You can then grep the ObjectId. For the complete solution, please check the link above.

Update by Id not finding document

I am attempting to perform a relatively simple update using Mongoose:
var location = locationService.getLocation(typedLat, typedLong, travelRadius);
DocumentModel.update({_id : passedInId }, { location : location }, function(err, resp){
next(err, resp);
});
In this case passedInId is the string:
"561ee6bbe4b0f25b4aead5c8"
And in the object (test data I created) the id is:
"_id": ObjectId("561ee6bbe4b0f25b4aead5c8")
When I run the above however, matched documents is 0. I assume this is because passedInId is being treated like a string, however when I type it to an ObjectId:
var queryId = ObjectId(passedInId)
The result is the same, the document doesn't match. What am I missing? This seems like it should be obvious....
Mongoose will correctly interpret a string as an ObjectId. One of the following must be the case:
That record is not in the collection. Run a query in the mongo shell to check.
Mongoose I'd looking in collection other than the one containing your test data. Remember, by default, mongo will lowercase the name under which you register your model and will add an a "s" to it.
Lastly, and your answer speaks to this, maybe your model it's just not being updated with any new information.
This behavior was because I had not yet updated the model in mongoose to include the location element. There is no error, it just doesn't match or do anything.

Argument passed in must be a single String of 12 bytes

mongoDB collection contains the following data
db.stack.find()
{ "_id" : "8GieRu" }
The _id is not single String of 12 bytes,
As per the MongoDB document of [ObjectID][1], id (string) – Can be a 24 byte hex string, 12 byte binary string or a Number.
Using Mongoose this collection is accessed using this Json
{"_id" : new mongoose.Types.ObjectId("8GieRu")}
and throws the below error
/node_modules/mongoose/node_modules/mongodb/node_modules/bson/lib/bson/objectid.js:35
throw new Error("Argument passed in must be a single String of 12 bytes or
^
Error: Argument passed in must be a single String of 12 bytes or a string of 24 hex characters
at new ObjectID (/node_modules/mongoose/node_modules/mongodb/node_modules/bson/lib/bson/objectid.js:35:11)
[1]: http://mongodb.github.io/node-mongodb-native/api-bson-generated/objectid.html
Mongoose is strictly checking the ObjectId of fixed length, how can i pass Object_id using mongoose with the given length
You mix two concepts here.
While "_id" can have any value (even subdocument like {firstName:'Foo',lastName:'Simpson'}, "ObjectId" has a fixed set of types with some restrictions, as the error message correctly states.
So your statement should be
{'_id':'putWhatEverYouWantHere'}
I had the problem in my router order:
app.get('/jobs', controllers.jobs.getAllJobs);
app.get('/jobs/karriere', controllers.jobs.getAllJobsXML);
app.get('/jobs/:id', controllers.jobs.getJob);
app.get('/jobs/:id/xml', controllers.jobs.getJobXML);
I defined /jobs/karriere after /jobs/:id so the application thought that "karriere" was an ObjectID and returned the error. The code above is the working one.
Make sure the method you are using in client and server side match. This error also shows when you have e.g. GET being sent from client side and POST required on the server side.
You are passing any
ObjectID undefinded
If the ObjectID is undfined thaen this error will come.
In my case, I'm using mongoose. and I am not able to query with something like this:
{ "_id" : "8GieRu" }
Until I bumped into my model file and specified this line counter.model.js
var CounterSchema = new Schema({
_id: String,
sequence_value: Number
})
Notice that I specified the data type as string for _id in my model.
and the, in my query, I didn't need to convert string to ObjectId.
Query now works as simple as what the filter:
{ "_id" : "8GieRu" }
same problem faced by me but after a RND . I identified that i passed wrong {Id:Undefined} so problem occured so please firstly check your Id which you passed in URL.
Error = "http://localhost:4000/api/deleteBook/Undefined"
Right = "http://localhost:4000/api/deleteBook/5bb9e79df82c0151fc0cd5c8"

Resources