As I understand it, when you enter something into GridFS it gets entered into 2 different collections under the hood. One for the raw chunks of data and one for the meta data files.
Also from what I understand from MongoDB's documentation is that you can only retrieve a document from GridFS with an id or name.
var gs = new mongodb.GridStore(db, "test.png", "w", {
"content_type": "image/png",
"metadata":{
"author": "Daniel"
},
"chunk_size": 1024*4
});
So what if I want to get a subset of documents from GridFS? For example what if I want all GridStores with:
metadata: {author: "Daniel"}
Why can't I use standard mongo queries { field: somevalue } and retrieve documents that way?
Does anybody know how this can be done? I'm using the javascript API on node.js.
You can query the db.files collection just like any other collection:
db.collection('fs.files')
.find({ 'metadata.author' : 'Daniel' })
.toArray(function(err, files) {
if (err) throw err;
files.forEach(function(file) {
var gs = new mongodb.GridStore(db, file._id, 'r');
...
});
});
Although instead of plain forEach you may want to use async.each or any of the other async.* methods.
Related
Given the data structure below in firebase, i want to run a query to retrieve the blog 'efg'. I don't know the user id at this point.
{Users :
"1234567": {
name: 'Bob',
blogs: {
'abc':{..},
'zyx':{..}
}
},
"7654321": {
name: 'Frank',
blogs: {
'efg':{..},
'hij':{..}
}
}
}
The Firebase API only allows you to filter children one level deep (or with a known path) with its orderByChild and equalTo methods.
So without modifying/expanding your current data structure that just leaves the option to retrieve all data and filter it client-side:
var ref = firebase.database().ref('Users');
ref.once('value', function(snapshot) {
snapshot.forEach(function(userSnapshot) {
var blogs = userSnapshot.val().blogs;
var daBlog = blogs['efg'];
});
});
This is of course highly inefficient and won't scale when you have a non-trivial number of users/blogs.
So the common solution to that is to a so-called index to your tree that maps the key that you are looking for to the path where it resides:
{Blogs:
"abc": "1234567",
"zyx": "1234567",
"efg": "7654321",
"hij": "7654321"
}
Then you can quickly access the blog using:
var ref = firebase.database().ref();
ref.child('Blogs/efg').once('value', function(snapshot) {
var user = snapshot.val();
ref.child('Blogs/'+user+'/blogs').once('value', function(blogSnapshot) {
var daBlog = blogSnapshot.val();
});
});
You might also want to reconsider if you can restructure your data to better fit your use-case and Firebase's limitations. They have some good documentation on structuring your data, but the most important one for people new to NoSQL/hierarchical databases seems to be "avoid building nests".
Also see my answer on Firebase query if child of child contains a value for a good example. I'd also recommend reading about many-to-many relationships in Firebase, and this article on general NoSQL data modeling.
Given your current data structure you can retrieve the User that contains the blog post you are looking for.
const db = firebase.database()
const usersRef = db.ref('users')
const query = usersRef.orderByChild('blogs/efg').limitToLast(1)
query.once('value').then((ss) => {
console.log(ss.val()) //=> { '7654321': { blogs: {...}}}
})
You need to use limitToLast since Objects are sorted last when using orderByChild docs.
It's actually super easy - just use foreslash:
db.ref('Users').child("userid/name")
db.ref('Users').child("userid/blogs")
db.ref('Users').child("userid/blogs/abc")
No need of loops or anything more.
I am using Parse Dashboard for User Management of my iOS Application.Also, I am using external APIs which are using MongoDB database.
The issue currently I am facing is the User created from Parse Dashboard is having small id instead of MongoDB's ObjectID, and other resources which are not over parse are generated by normal ObjectID.
eg. User Object:
{
_id:"qVnyrGynJE",
user_name:"Aditya Raval"
}
Document Object:
{
_id:"507f191e810c19729de860ea",
doc_name:"Marksheet",
user:"qVnyrGynJE"
}
Task Object:
{
_id:"507f191e810c19729de860ea",
task_name:"Marksheet",
user:"qVnyrGynJE"
}
I am also using Keystone.js as a Backend Admin Dashboard.So basically due to this mix kind of IDs relationships inside KeyStone.js is broken and Keystone.js gets crashed.
So I want to migrate all my existing small IDs to normal MongoDB ObjectIDs without breaking into relationships or any other walkthrough by fixing Keystone.js
You can run something like this:
var users = db.Users.find({});
for(var i = 0; i < users.length(); i++)
{
var oldId = users[i]._id;
delete users[i]._id;
db.Users.insert(users[i], function(err, newUser) {
db.Documents.updateMany({"user": oldId},{ $set: { "user": newUser._id }});
//Do this for all collections that need to be update.
});
);
}
db.Users.deleteMany({_id: { $type: "string" }});
So I have a model that is for a recipe where it has a relation of 'ingredients' and that is just an array of ObjectIds. When I run the following query on mongo shell it works fine and returns all my data.
Example model :
{
"name": "...",
"_id": ObjectId("530ca903746515c0161e6b9f"),
"ingredients": [
ObjectId("53069363ff7447a81a3a7a1d"),
ObjectId("53069363ff7447a81a3a7a17")
]
}
Query:
db.drink.find({"ingredients":{"$in":[ObjectId("53069364ff7447a81a3a7a87"), ObjectId("530fb948c1a3ff480d58e43c")]}});
Using sails.js though their waterline orm, they don't really have a way to query this though or at least through any possible google search that I can find. So trying to use the native driver I have something like the following -
var ings = new Array();
for (var i in req.body) {
ings.push(new ObjectID(req.body[i].toString()));
}
Drink.native(function(err, collection){
if(err){
return res.send(err, 500);
} else {
collection.find({"ingredients":{"$in":ings}}).toArray(function(err, data){
console.log(data);
});
}
});
The thing is the data array returned in the callback is always empty. If I check the 'ings' array it is an array of objectids so I am not sure why it won't return any data. If I remove the json object in the 'find' function it does return everything. Anyone have any idea how to make this query return data when using sails.js?
The code above actually is working it was an internal data issue on the mongodb where IDs were not matching between relations. When the relations were rebuilt all is working as it should be.
I have a Problem in with my REST API. It will not correctly communicate with mongodb. Node.js throw following Exception:
Error: Argument passed in must be a single String of 12 bytes or a string of 24 hex characters
My Manager looks like follows:
app.get('/users/:id', function(req, res) {
userProvider.fetchUserById(req.params.id, function(error, user) {
if (user == null) {
res.send(error, 404);
} else {
res.send(user);
}
});
});
My Provider looks like follows:
this.fetchUserById = function(id, cb) {
this.db.collection(usersTable, function(error, users) {
if (error) {
cb(error, null);
} else {
users.findOne({
_id:users.db.bson_serializer.ObjectID.createFromHexString(id)
}, function(error, result) {
cb(error, result);
});
}
});
};
My Mongoobject looks like follows:
{
"title": "title",
"givenname": "Vorname",
"lastname": "Nachname",
"name": "Vorname Nachname",
"username": "Benutzername",
"password": "Kennwort",
"email": "kuerzel#emailadresse.de",
"phone": "0000 - 000000",
"fax": "000000 - 000000",
"lastlogin": "15.11.2013",
"createdate": "15.11.2013",
"changedate": "15.11.2013",
"state": "1",
"_id": ObjectId("5283fbf56e3adb01f1000001")
}
When I now send a GET: /users/1 the error will thrown.
I´am new in Node.js. :-)
If your human readable IDs are unique, you can use them.
Just insert the documents with your nice id:
db.users.insert({ _id: myVeryNiceUniqueId, title: '...' }, ...);
As alternative, you could add an additional 'external id' to the document and create an unique index. But you will get many, many problems when you want to scale (sharding). Who is going to create the unique IDs? You will have something not-so-human-readable, I promise.
Edit
In Mysql I have start the Ids from 5XXXX and used them. How can I realize this with mongodb
You should ask another question or search answers. It is not a trivial task and you have to take concurrency into account. Maybe the concept of findAndModify or "Upsert" could help you. One of the most important things is that you need a way of atomic updates for the ID-generator. In one project in past I used Redis for this, but I'd always use MongoDB IDs since then. You should know that MongoDB ObjectIDs usually are generated by the Client-Driver. For REST services this is ideal: save document to DB and return status 201 Created, url=.../user/98357348957.
btw. should I generate the OrderIDs in Javascript by checking the highest value of an external id in the collection?
As said, you can get into trouble with usage of duplicate IDs when your server is in high contention. Try to find a solution with findAndModify or cache a block of IDs and use them in one process. I probably wouldn't search max and increment but use a special collection for ID-management.
What is here best practice?
Don't know. Using an ID-Generator. Depends on your enviroment (single process, single machine). If it is a simple setup, you'll find simple solutions. The setup for Twitter is not simple and they have developed a solution called twitter / snowflake. If you really want very nice, short IDs, but you have more than one process/machine, Redis might be want you want. And if the ID doesn't have to be contigous (without small gaps), you could draw a bunch of new IDs (e.g 100) and cache them for future consumption per process.
Can I hash the objectID to an 5 digit number?
No, there is no guarantee that this number is unique.
In my application im using node.js with redis database.How can i save many json objects in redis.
db.save({
description:'sdsd',userId:'324324',url:'http://www.abc.com/abc.html',appId:'123456'
}, function (err, res) {
if(err){
console.log(err);
return;
}
else{
console.log(res);
}
});
In couch db we can save the json objects again and again as document by using the above code.How to do this in redis.From their documentation i came to know below code save the json objects
client.hmset("hosts", "mjr", "1", "another", "23", "home", "1234");
Again i want to save the other json object in same "hosts" like below
client.hmset("hosts", "mjr", "2", "another", "33", "home", "1235");
How can i do this.
Redis storage model is different from CouchDB. In Redis, everything gets accessed by its key, so it all depends how you plan to retrieve your data.
So if you'd like to be able to retrieve data by userId, use this as the key.
redis.set('324324', {
description:'sdsd',
url:'http://www.abc.com/abc.html',
appId:'123456'
});
But if you need to retrieve a piece of data using more than one piece of the data, then redis may not be suitable.
In some cases, you may be able to use some tricks, so that to be able to query on both userId and appId, you could use 324324:123456 as the key, and query using
GET 324324:*
to get all apps for one user
or
GET *:123456
to get all users for a given app.