I have an app that stores user uploaded spreadsheets as tables in PostgreSQL. Everytime an user uploads a spreadsheet I create a record in a Dataset table containing the physical table name, its alias and the owner. I can retrieve a certain Dataset information with
GET domain.com/v1/Datasets/{id}
AFAIK, the relation between rows in Dataset and physical tables can't be enforced by a FK, or at least I haven't seen anyones creating FKs on the information_schema of PostgreSQL, and FKs can't drop tables, or can they? So it's common to have orphan tables, or records in Dataset that point to tables that no longer exist. I have managed this with business logic and cleaning tasks.
Now, to access one of those physical tables, for example one called nba_teams I would need to declare an NbaTeams model in loopback and restart the app, then query its records with
GET domain.com/v1/NbaTeams/{id}
But that can't scale, specially if I'm already having like 100 uploads a day. So from where I'm standing, there are two ways to go:
1.- Create one model, then add 4 custom methods that accepts a table name as a string, and perform the next CRUD operation on that table name via raw queries. For example, to list the records:
GET domain.com/v1/Datasets/getTable/NbaTeams
or, to update one team
PUT domain.com/v1/Datasets/getTable/NbaTeams/{teamId}
This sounds unelegant but should work.
2.- Create a custom method that accepts a table name as a string, which in turn creates an ephemeral model and forward the HTTP verb and the rest of the arguments to it
dataSource.discoverAndBuildModels('nba_teams', {
owner: 'uploader'
}, function (err, models) {
console.log(models);
models.NbaTeams.find(function (err, act) {
if (err) {
console.error(err);
} else {
console.log(act);
}
dataSource.disconnect();
});
});
this second one I haven't got to work yet, and I don't know how much overhead it might have, but I'm sure it's doable.
So before I dig in deeper I came to ask: has anybody dealt with this row-to-table relation? What are the good practices in this?
In the end, I did my own hacky workaround and I thought it may help someone, some day.
What I did was put a middleware (with regular express syntax) to listen for /v1/dataset{id_dataset} , create the model on the fly and pass the execution to the next middleware
app.use('/v1/dataset:id_dataset', function(req, res, next) {
var idDataset=req.params.id_dataset;
app.getTheTable(idDataset,function(err,result) {
if(err) {
console.error(err);
res.json({"error":"couldn't retrieve related table"});
} else {
next();
}
});
});
inside the app.getTheTable function, I'm creating a model dynamically and setting it up before callback
app.getTheTable = function (idDataset, callback) {
var Table = app.models.Dataset,
modelName='dataset'+idDataset,
dataSource;
Table.findById(idDataset, function (err, resultados) {
if (err) {
callback(new Error('Unauthorized'));
} else {
if(app.models[modelName]) {
callback(null,modelName); // model already exists
} else {
var theDataset = dataSource.createModel(modelName, properties, options);
theDataset.settings.plural = modelName;
theDataset.setup();
app.model(theDataset);
var restApiRoot = app.get('restApiRoot');
app.use(restApiRoot, app.loopback.rest());
callback(null, modelName);
}
}
});
};
It's hacky, I know, and I believe there must be some kind of performance penalty for overloading restApiRoot middleware, but it's still better tan creating 500 models on startup to cover all possible dataset requests.
Related
I am building an API and came across an issue that I have a few ideas of how to solve, but I was wondering what is the most optimal one. The issue is the following:
I have a Product model which has, for the sake of simplicity one field called totalValue.
I have another model called InventoryItems, which, whenever is updated, the totalValue of Product must also be updated.
For example, if the current totalValue of a product is say $1000, when someone purchases 10 screws at a cost of $1 each, a new InventoryItem record will be created:
InventoryItem: {
costPerItem: 1,
quantity: 10,
relatedToProduct: "ProductXYZ"
}
At the same time of creation of that item, totalValue of the respective ProductXYZ must be updated to now $1100.
The question is what is the most efficient and user-friendly way to do this?
Two ways come to my mind (and keep in mind that the code bellow is kinda pseudo, I have intentionally omitted parts of it, that are irrelevant for the problem at hand):
When the new InventoryItem is created, it also queries the database for the product and updates it, so both things happen in the same function that creates the inventory item:
function async createInventoryItem(req, res) {
const item = { ...req.body };
const newInventoryItem = await new InventoryItem({...item}).save();
const foundProduct = await Product.find({ name: item.relatedtoProduct }).exec();
foundProduct.totalValue = foundProduct.totalValue + item.costPerItem * item.quantity;
foundProduct.save();
res.json({ newInventoryItem, newTotalOfProduct: foundProduct.totalValue });
}
That would work, my problem with that is that I will no longer have "a single source of truth" as that approach will make it hard to update the code, as updating a given Product will be scattered all over the project.
The second approach that comes to my mind is that, when I receive the request to create the item, I do create the item, and then I make an internal request to the other endpoint that handles product updates, something like:
function async createInventoryItem(req, res) {
const item = { ...req.body };
const newInventoryItem = await new InventoryItem({...item}).save();
const totalCostOfNewInventoryItem = item.costPerItem * item.quantity;
// THIS is the part that I don't know how to do
const putResponse = putrequest("/api/product/update", {
product: item.relatedtoProduct,
addToTotalValue: totalCostOfNewInventoryItem
});
res.json({ newInventoryItem, newTotalOfProduct: putResponse.totalValue });
}
This second approach solves the problem of the first approach, but I don't know how to implement it, and it is I'm guessing a form of requests chaining or rerouting? Also I am guessing that the second approach will not have a performance penalty, since node will be sending requests to itself, so no time lost in accessing servers across the world or whatever)
I am pretty sure that the second approach is the one that I have to take (or is there another way that I am currently not aware of??? I am open to any suggestions, I am aiming for performance), but I am unsure of exactly how to implement it.
I'm using FeathersJS and MongoDB to develop an app. I want to prevent some services to create duplicates of some values (or pairs of values).
For example, the FeathersJS "Authenticate" service created with the feathers-cli tool doesn't prevent the app from creating 2 or more users with the same email (at least using MongoDB). Another example would be a service to create some "categories" for each user. I want that the backend prevents a user to create 2 or more categories with the same name, but I need to allow 2 different users to create their own categories although their names are the same (but not the users).
I know I can do this by using indexes in the MongoDB collections, but this would make the app MongoDB dependant.
Is there someone that knows if there's any kind of hook or whatever that is the recommended way to do such things "the FeathersJS way"?
Thank you!
In most cases uniqueness can - and should - be insured at the database or ORM/ODM level since it will give you the best performance (something that in most cases isn't worth sacrificing for portability).
A more Feathers-y way and to accomplish more complex restrictions would be Feathers hooks which are an important part of Feathers and explained in detail in the basics guide.
In this case, a before hook could query the total of items and throw an error if there are any:
const { Conflict } = require('#feathersjs/errors');
app.service('myservice').hooks({
before: {
create: [async context => {
const { fieldA, fieldB } = context.data;
// Request a page with no data and extract `page.total`
const { total } = await context.service.find({
query: {
fieldA,
fieldB,
$limit: 0
}
});
if(total > 0) {
throw new Conflict('Unique fields fieldA and fieldB already exist');
}
return context;
}]
}
})
Question: When creating something like a simple many to many friendship in mongoose, I know how to create it on ONE object, for instance, the code below in the controller shows that I am finding one user, and pushing to his friends array another user, being referenced via ObjectId.
In this way, when I look at the Json file, I can see user with _id of "57ed2e8c9cf3083c2ccec173", has a new friend in his friend's array, and I can run a population to get that friend user document. However, user who was added as a friend does not have these capabilities because his array of friends is still empty.
I know there are multiple ways to go about this, as I have read the docs, which say I could simply now push user 1 into user 2's friends array, but, in the words of the docs: "It is debatable that we really want two sets of pointers as they may get out of sync. Instead we could skip populating and directly find() the stories we are interested in."
In other words, if you have an event model with many users, and user model with many events, and you need to access the array of users from the event document, and the array of events from the user document... Would it be best to just push each instance into each other?
Is this the correct way of thinking?
Thanks
```
app.post('/friendships', function(req, res) {
User.findOne({
_id: "57ed2e8c9cf3083c2ccec173"
}, function(err, user1) {
User.findOneAndUpdate({
_id: "57ed2ebbedcd96a4536467f7"
}, {$push: {friends: user1 }}, {upsert: true}, function(err, user2) {
console.log("success");
})
})
});
```
Yes, this is the correct way of thinking, considering the limitations of Mongo for that sort of data.
When you store such an information in two places, you need to make sure that it is consistent - i.e. either it is present in both places or not. You don't have transactions in Mongo so the only way you can do it is to chain the requests and manually roll back the first one if the second one failed, hoping that it's possible to do (which may not be the case - if the second update failed because you lost a connection to the database, there is a good chance that your rollback will fail as well, in which case your database is left in an inconsistent state).
An alternative would be to store only one half of the relationship - e.g. only store events in users, but no users in events, using your example. That way the data would be consistently stored in one place but then if you wanted to get a list of users for a certain event, you'd have to make a possibly expensive database lookup instead of having it already present in the event document.
In practice in most cases I have seen storing data in two places and trying to keep them consistent.
Though it is usually done with storing documents IDs, so instead of:
{$push: {friends: user1}}
it's usually:
{$push: {friends: user1._id}}
(or just using the _id if you have it in the first place)
And instead of $push you can use $addToSet - see: https://docs.mongodb.com/manual/reference/operator/update/addToSet/
Here is a basic concept of adding a two-directional friendship between id1 and id2:
function addFriendship(id1, id2) {
User.findOneAndUpdate({_id: id1}, {$addToSet: {friends: id2}}, err => {
if (err) {
// failure - no friendship added
} else {
// first friendship added, trying the second:
User.findOneAndUpdate({_id: id2}, {$addToSet: {friends: id1}}, err => {
if (err) {
// second friendship not added - rollback the first:
User.findOneAndUpdate({_id: id1}, {$pull: {friends: id2}}, err => {
if (err) {
// we're screwed
} else {
// rolled back - consistent state, no friendship
}
});
} else {
// success - both friendships added
}
});
}
});
}
Not pretty and not bulletproof but that's the most you can hope for with a database with no transactions where denormalized data is the norm.
(Of course friendship don't always work that way that they have to be bidirectional, but this is just an example of a pattern that is common for any many-to-many relationaship.)
I'm trying to limit the fields a user can post when inserting an object in mongodb. I know ho i can enforce fields to be filled but I can't seem to find how to people from inserting fields that I don't want.
This is the code I have now for inserting an item.
app.post("/obj", function (req, res) {
var newObj = req.body;
//TODO filter fields I don't want ?
if (!(newObj .id || newObj .type)) {
handleError(res, "Invalid input", "Must provide a id and type.", 400);
return;
}
db.collection(OBJ_COLLECTION).insertOne(newObj, function(err, doc) {
if (err) {
handleError(res, err.message, "Failed to create new object.");
} else {
res.status(201).json(doc.ops[0]);
}
});
});
There's likely JS native ways to do this, but I tend to use Lodash as my toolbox for most projects, and in that case what I normally do is setup a whitelist of allowed fields, and then extract only those from the posted values like so:
const _ = require('lodash');
app.post("/obj", function (req, res) {
var newObj = _.pick(req.body, ['id', 'type','allowedField1','allowedField2']);
This is pretty straightforward, and I usually also define the whitelist somewhere else for reuse (e.g. on the model or the like).
As a side note, I avoid using 'id' as a field that someone can post to for new objects, unless I really need to, to avoid confusion with the autogenerated _id field.
Also, you should really look into mongoose rather than using the straight mongodb driver, if you want to have more model-based control of your documents. Among other things, it will strip any fields off the object if they're not defined in the schema. I still use the _.pick() method when there are things that are defined in the schema, but I don't want people to change in a particular controller method.
I'm working with sails.js and mongo.db following some tutorials and creating a custom application and things are going well. Largely I'm using the built in, backbone I believe, scrud functions, I'm wondering how I could create a database entry from scratch. For example, the following works great from form data in my Student Controller:
create: function (req, res, next) {
Student.create(req.params.all(), function studentCreated(err, student) {
if (err) {
console.log(err);
req.session.flash = {
err: err
}
return res.redirect('/');
}
res.redirect('/');
});
},
For simplicity my Student model currently just has a first name and a one way association with a schools model.
module.exports = {
attributes: {
school: {
model: 'school'
},
first_name: {
type: 'string'
},
},
};
Say I wanted to, for the sake of understanding, just create a student with a fixed first name "Bob" and fixed school Id "xyz" in another action, without using the built in backbone create functions, nor using defaultTo in the model, nor using any form data. I would like to just code creating a database entry in an action, for example a test action in my Student controller. How would I go about this? I tried googling this a little, but given it's beginner nature I'm sure my query parameters were not particularly good.
If I correctly understand the question, I think you've already done most of the work. In the Student.create() call, replace req.params.all() by {first_name:'Bob', school:'xyz'}.
If you want to mix it with values from the request params, you could use {first_name:'bob', school: req.param('school')}