Given the data structure below in firebase, i want to run a query to retrieve the blog 'efg'. I don't know the user id at this point.
{Users :
"1234567": {
name: 'Bob',
blogs: {
'abc':{..},
'zyx':{..}
}
},
"7654321": {
name: 'Frank',
blogs: {
'efg':{..},
'hij':{..}
}
}
}
The Firebase API only allows you to filter children one level deep (or with a known path) with its orderByChild and equalTo methods.
So without modifying/expanding your current data structure that just leaves the option to retrieve all data and filter it client-side:
var ref = firebase.database().ref('Users');
ref.once('value', function(snapshot) {
snapshot.forEach(function(userSnapshot) {
var blogs = userSnapshot.val().blogs;
var daBlog = blogs['efg'];
});
});
This is of course highly inefficient and won't scale when you have a non-trivial number of users/blogs.
So the common solution to that is to a so-called index to your tree that maps the key that you are looking for to the path where it resides:
{Blogs:
"abc": "1234567",
"zyx": "1234567",
"efg": "7654321",
"hij": "7654321"
}
Then you can quickly access the blog using:
var ref = firebase.database().ref();
ref.child('Blogs/efg').once('value', function(snapshot) {
var user = snapshot.val();
ref.child('Blogs/'+user+'/blogs').once('value', function(blogSnapshot) {
var daBlog = blogSnapshot.val();
});
});
You might also want to reconsider if you can restructure your data to better fit your use-case and Firebase's limitations. They have some good documentation on structuring your data, but the most important one for people new to NoSQL/hierarchical databases seems to be "avoid building nests".
Also see my answer on Firebase query if child of child contains a value for a good example. I'd also recommend reading about many-to-many relationships in Firebase, and this article on general NoSQL data modeling.
Given your current data structure you can retrieve the User that contains the blog post you are looking for.
const db = firebase.database()
const usersRef = db.ref('users')
const query = usersRef.orderByChild('blogs/efg').limitToLast(1)
query.once('value').then((ss) => {
console.log(ss.val()) //=> { '7654321': { blogs: {...}}}
})
You need to use limitToLast since Objects are sorted last when using orderByChild docs.
It's actually super easy - just use foreslash:
db.ref('Users').child("userid/name")
db.ref('Users').child("userid/blogs")
db.ref('Users').child("userid/blogs/abc")
No need of loops or anything more.
Related
I am struggling to find the solution for that.
I want to have users which can belong to many organizations.
Each user can have a different role (I would prefer even roles but it sounds even more complicated...) at a specific organization.
In the table like User_Organization_Role I need to have fields like role (roleId?), isActive. Maybe some more.
I am using Feathers Plus generator but I do not think it matters in this case, however it may be beneficial to add something to the schema file?
I thought having belongsTo with simple organizationId field will be sufficient but I've realized that changing that to manyToMany, later on, would be painful so I think it is much better to implement that now.
I will appreciate any solutions / suggestions / best practices etc.
n:m relations are by far the most difficult to handle, and there's really no one-size-fits-all solution. The biggest thing is to read and understand this page and its sub-pages, and then read them 2 more times for good measure. Try to focus on doing one thing at a time. I outline how I would approach this with feathersjs in this issue:
https://github.com/feathersjs/feathers/issues/852#issuecomment-406413342
The same technique could be applied in any application... the basic flow goes like this:
Create or update your primary objects first (users, organizations, roles, etc.). There are no relations made at this point. You need to have your objects created before you can make any relations.
Create or update the relations. This involves updating a "join" table (aka: "mapping" or "through" table) with data from step #1. The join table can (and should) have its own model. It should contain a foreign key for each of the objects you are associating (userId, organizationId, roleId etc.). You can put other fields in this table too.
Here is some pseudo code for how I would define my models (only showing relevant code for brevity). There is a little more to it than what I describe below, but this should get you started.
const UserOrganizationRole = sequelize.define('User_Organization_Role', {
// Define any custom fields you want
foo: DataTypes.STRING
})
// Let sequelize add the foreign key fields for you.
// Also, save a reference to the relationship - we will use it later
User.Organization = User.belongsToMany(Organization, { through: UserOrganizationRole });
User.Role = User.belongsToMany(Role, { through: UserOrganizationRole });
Organization.User = Organization.belongsToMany(User, { through: UserOrganizationRole });
Role.User = Role.belongsToMany(User, { through: UserOrganizationRole });
... and here is how I would go about handling inserts
const user = await User.create({ ... });
const org = await Organization.create({ ... });
const role = await Role.create({ ... });
await UserOrganizationRole.create({
userId: user.id,
organizationId: org.id,
roleId: role.id,
foo: 'bar'
});
... and finally, load the data like so:
// Now we can reference those relationships we created earlier:
const user = await User.findById(123, {
include: [User.Organization, User.Role]
});
const org = await Organization.findById(456, {
include: [Organization.User]
});
I have been working with the google cloud library, and I can successfully save data in DataStore, specifically from my particle electron device (Used their tutorial here https://docs.particle.io/tutorials/integrations/google-cloud-platform/)
The problem I am now having is retrieving the data again.
I am using this code, but it is not returning anything
function getData(){
var data = [];
const query = datastore.createQuery('ParticleEvent').order('created');
datastore.runQuery(query).then(results => {
const event = results[0];
console.log(results);
event.forEach(data => data.push(data.data));
});
console.log(data)
}
But each time it is returning empty specifically returning this :
[ [], { moreResults: 'NO_MORE_RESULTS', endCursor: 'CgA=' } ]
, and I can't figure out why because I have multiple entities saved in this Datastore.
Thanks
In the tutorial.js from the repo mentioned in the tutorial I see the ParticleEvent entities are created using this data:
var obj = {
gc_pub_sub_id: message.id,
device_id: message.attributes.device_id,
event: message.attributes.event,
data: message.data,
published_at: message.attributes.published_at
}
This means the entities don't have a created property. I suspect that ordering the query by such property name is the reason for which the query doesn't return results. From Datastore Queries (emphasis mine):
The results include all entities that have at least one value for
every property named in the filters and sort orders, and whose
property values meet all the specified filter criteria.
I'd try ordering the query by published_at instead, that appears to be the property with a meaning closest to created.
I am running an iOS app where I display a list of users that are currently online.
I have an API endpoint where I return 10 (or N) users randomly, so that you can keep scrolling and always see new users. Therefore I want to make sure I dont return a user that I already returned before.
I cannot use a cursor or a normal pagination as the users have to be returned randomly.
I tried 2 things, but I am sure there is a better way:
At first what I did was sending in the parameters of the request the IDs of the user that were already seen.
ex:
But if the user keeps scrolling and has gone through 200 profiles then the list is long and it doesnt look clean.
Then, in the database, I tried adding a field to each users "online_profiles_already_sent" where i would store an array of the IDs that were already sent to the user (I am using MongoDB)
I can't figure out how to do it in a better/cleaner way
EDIT:
I found a way to do it with MySQL, using RAND(seed)
but I can't figure out if there is a way to do the same thing with Mongo
PHP MySQL pagination with random ordering
Thank you :)
I think the only way that you will be able to guarentee that users see unique users every time is to store the list of users that have already been seen. Even in the RAND example that you linked to, there is a possibility of intersection with a previous user list because RAND won't necessarily exclude previously returned users.
Random Sampling
If you do want to go with random sampling, consider Random record from MongoDB which suggests using an an Aggregation and the $sample operator. The implementation would look something like this:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
const randomDocs = await collection
.aggregate([{
$sample: {
size: 5
}
}])
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
});
randomDocs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
client.close();
}());
Cache of Previous Users
If you go with maintaining a list of previously viewed users, you could write an implementation using the $nin filter and store the _id of previously viewed users.
Here is an example using a weather database that I have returning entries 5 at a time until all have been printed:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
let previousEntries = [], // Track ids of things we have seen
empty = false;
while (!empty) {
const findFilter = {};
if (previousEntries.length) {
findFilter._id = {
$nin: previousEntries
}
}
// Get items 5 at a time
const docs = await collection
.find(findFilter, {
limit: 5,
projection: {
main: 1
}
})
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
})
.toArray();
// Keep track of already seen items
previousEntries = previousEntries.concat(docs.map(doc => doc.id));
// Are we still getting items?
console.log(docs.length);
empty = !docs.length;
// Print out the docs
docs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
}
client.close();
}());
I have encountered the same issue and can suggest an alternate solution.
TL;DR: Grab all Object ID of the collections on first landing, randomized using NodeJS and used it later on.
Disadvantage: slow first landing if have million of records
Advantage: subsequent execution is probably quicker than the other solution
Let's get to the detail explain :)
For better explain, I will make the following assumption
Assumption:
Assume programming language used NodeJS
Solution works for other programming language as well
Assume you have 4 total objects in yor collections
Assume pagination limit is 2
Steps:
On first execution:
Grab all Object Ids
Note: I do have considered performance, this execution takes spit seconds for 10,000 size collections. If you are solving a million record issue then maybe used some form of partition logic first / used the other solution listed
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id; });
OR
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id.valueOf(); });
Result:
ObjectId("FirstObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ThirdObjectID"),
ObjectId("ForthObjectID"),
Randomized the array retrive using NodeJS
Result:
ObjectId("ThirdObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ForthObjectID"),
ObjectId("FirstObjectID"),
Stored this randomized array:
If this is a Server side script that randomized pagination for each user, consider storing in Cookie / Session
I suggest Cookie (with timeout expired linked to browser close) for scaling purpose
On each retrieval:
Retrieve the stored array
Grab the pagination item, (e.g. first 2 items)
Find the objects for those item using find $in
.
db.getCollection('my_collection')
.find({"_id" : {"$in" : [ObjectId("ThirdObjectID"), ObjectId("SecondObjectID")]}});
Using NodeJS, sort the retrieved object based on the retrived pagination item
There you go! A randomized MongoDB query for pagination :)
I have a task object, the belongs to a job object. It is a belongsTo() relationship, but I'm using a hasOne() relationship because I just couldn't figure out belongsTo().
My models are:
var job = Bookshelf.Model.extend({
tableName: 'jobs',
idAttribute: 'id'
});
var task = Bookshelf.Model.extend({
tableName: 'tasks',
idAttribute: 'id',
job: function() {
return this.hasOne(Job);
},
job_summary: function() {
return this.job().fetch().get('summary');
}
});
I've tried a few combinations of the .job_summary() function, but nothing seems to work. When I create a task object, I just want to be able to get the job summary of the related task, like:
var task_jobsummary = task.job_summary();
What am I missing?
EDIT: I actually have 2 hasOne() targets I need to get data for.
EDIT: I think my problem is not understanding promises. If I were doing this in PHP, I'd do something like:
$myvar = stdClass();
$myvar->somefield = $task->getSomefield();
$myvar->anotherfield = $task->getAnotherField();
$myvar->jobfield = $job->getSomeJobField();
// ... etc
I want to simple return values from a then() after a fetch().
EDIT: Here is my code from my router:
var route_task(req, res, next) {
res.locals.username = 'debug_username';
res.locals.userpic = '/img/tmpuserpic.png';
new Model.Task().fetch({id: req.params.id, withRelated: ['job']})
.then(function(task) {
task.related('job').fetch().then(function(job) {
res.locals.somefield = job.get('somefield');
});
})
.then(function() {
res.render('task');
})
}
The two vars before the Model are set, and my template sees them. However, the template doesn't see the res.locals.somefield setting.
You are a little bit confused, you need to have the belongsTo where you have the key and the hasOne where basically it is pointed.
For example a post has many (but is equal in concept) comments and a comment belongs to a post.
In the model you just need to say what relationship are, when you want to retrive it (maybe in the controller) you need to do something like this
new Model({id:4})
.fetchOne({withRelated: ['thing']})
.then(function(model) {
// retrive the relation
var things = model.things(); // you already had the query done
});
You need to elaborate this but with the concept in mind the documentation helps a lot
The best way I can describe what I'm after is by starting with a simplified demo page document:
{
name: "Page Name"
blocks: {
"top-content": 50f0ded99561a3b211000001,
"bottom-content": 50f0ded99561a3b211000002
}
}
Is there a way I can define this kind of many-to-many relation with mongoose so that I can look them up by string keys like top-content and bottom-content? The relational equivalent to this would be something like including a string on the many to many join table.
Important:
I don't want to embed the blocks I'm referencing as they could be referenced by multiple pages.
Mongoose's query population feature supports this, but top-content and bottom-content would need to be ObjectIds instead of strings (which is more efficient anyway).
After some chatting in #node.js on Freenode, it seems like I can just set this up as an array and put arbitrary key/value pairs containing the references I want.
At which point, I'll be dereferencing things myself.
{ //PAGE 1
name:"Sticks",
blocks:[
"top":[OID(5),OID(6)],
"bottom":[OID(7),OID(8)]
]
};
{ //PAGE 2
name:"Poopsicles",
blocks:[top:[OID(7)]]
};
//BLOCKS
{
OID:5,
name:"top"
};
{
OID:7,
name:"bottom rocker"
};
//QUERYING
page.findOne({name:"Sticks"}, function(e,d) {
var a = [];
for(l in d.blocks) {
a.append(d.blocks[l]);
}
blocks.find({oid:a},function (e,b) {
//do your stuff with the blocks and page here;
});
});