This is a really trivial problem. I am just curious on how to deal with this in a "professional" manner.
I am trying to stick to variable naming convention. For NodeJs I am doing camelCasing. For database, I am using PostgreSQL and using underscore_casing.
Now the problem arises when I query data from PostgreSQL. I'll get a user object with following format,
{user_id: 1, account_type : "Admin"}
I can pass this object directly to server side-render and will have to use underscore casing to access account_type. Of course, I can manually create a new user JSON object with property userId and accountType but that is unnecessary work.
Is it possible to follow variable naming convention for both language and avoid having mixed variable names casing in some files? What is a good way to stay organized?
The are two good ways to approach this issue. The simplest one - do no conversion, use the exact database names. And the second one is to camel-case columns automatically.
Either way, you should always follow the underscore notation for all PostgreSQL declarations, as it will give you the option to activate camel-casing in your app at a later time, if it becomes necessary. Never use camel-case inside the database, or you will end up in a lot of pain later.
If you want the best of both worlds, follow the underscore notation for all PostgreSQL declarations, and convert to camel-case as you read data.
Below is an example of how to do it properly with pg-promise, copied from event receive example:
// Example below shows the fastest way to camelize column names:
const options = {
receive(e) {
camelizeColumns(e.data);
}
};
function camelizeColumns(data) {
const template = data[0];
for (var prop in template) {
const camel = pgp.utils.camelize(prop);
if (!(camel in template)) {
for (var i = 0; i < data.length; i++) {
const d = data[i];
d[camel] = d[prop];
delete d[prop];
}
}
}
}
Also see the following article: Pg-promise and case sensitivity in column names.
UPDATE
The code above has been updated for use of pg-promise v11 or later.
I've struggled with this too, and I've concluded that there's really no way to avoid this kind of ugliness unless you rewrite the objects that come from the database. Fortunately, that's not too difficult in Javascript:
const fromDBtoJS = (obj) => {
// declare a variable to hold the result
const result = {};
// iterate over the keys on the object
Object.keys(obj).forEach((key) => {
// adjust the key
const newKey = key.replace(/_[a-z]/g, (x) => x[1].toUpperCase());
// add the value from the old object with the new key
result[newKey] = obj[key];
});
// return the result
return result;
};
Here's a JSFiddle. The "replace" code above was found here
If you wanted to use classes for models in your application, you could incorporate this code into the constructor or database load method so it's all handled more-or-less automatically.
Related
I have a strange error with mongoose deleteOne() function. Today I wanted to work on my project and got an error while deleting an item from a collection. It simply doesn't delete the document until I use a hardcoded parameter for the options object like this:
const { deletedCount } = await Model.deleteOne({symbol: 'hardcoded'})
// results in deletedCount = 1
But if I try to use a dynamic string like:
const test = 'dynamic'
const { deletedCount } = await Model.deleteOne({symbol: test})
// results in deletedCount = 0
It does no longer delete the document from my collection. The strange thing is yesterday it worked fine and deleted the item.
I tried one other thing I read regarding errors with deleteOne():
const { deletedCount } = await Model.deleteOne({symbol: JSON.stringifiy(symbol)})
But this doesn't work, too.
Does anyone have an idea what's going wrong?
I always default to using ids whenever possible to make sure there's no mistake in the data I am targeting with a given operation.
So in this case that would mean using findByIdAndDelete() instead.
If I don't know the id of the document I'm trying to delete, then only I'd use findOneAndDelete() or deleteOne(), as you have, with something other than an id to identify the document I'm looking for.
Are you certain that the key-value pair you're passing to the function exists in your database?
Problem solved. I accidentally added an additional space character at the end of the string. This is very strange because the error was there since the beginning of my project and yesterday it worked.
So for everyone who might have a similar problem:
I have a ejs template file where I render a html element like this:
<div id="<%= symbol %> ">
Then in my event handler for requesting the server to deleting one item from my list I use the id attribute as a request parameter in the body. In the route handler this parameter is passed to mongoose:
const { symbol } = req.body
const { deletedCount } = await Model.deleteOne({ symbol })
As I mentioned. In the template file after the last ejs seperator there is an addional space character that caused the error. I spotted this issue by making a copy of the monoogse query and than logged it to the console. There I could see the wrong condition parameter.
I am working with the below piece of code in Microsoft Bot Framework to access the list of regexes for global commands. This code is a part of botbuilder module:
if (typeof session.conversationData.globalCommands === "undefined") {
// An array which contains the list of all global commands
session.conversationData.globalCommands = [];
// Accessing the list of global commands
globalActions = session.library.actions.actions;
lenGlobalActions = Object.keys(globalActions).length;
// Assigning values to the above list
for (var i=0; i<lenGlobalActions; i++){
session.conversationData.globalCommands.push(globalActions[Object.keys(globalActions)[i]].options.matches);
}
}
// Checking if the incoming message from the user is a global command
var isGlobalCommand = session.conversationData.globalCommands.some(regex => regex.test(session.message.text));
The issue here is, the code runs fine for the first time and the values assigned to the variable session.conversationData.globalCommands are in the form given below:
However, after the first execution, the array converts to the below without any changes made in the code anywhere else:
Due to this, the line:
var isGlobalCommand = session.conversationData.globalCommands.some(regex => regex.test(session.message.text));
throws an exception "regex.test is not a function".
I am unable to understand why this should be happening and how do I solve this as I need this list separately for some processing.
I believe you cannot store complex types in the bot store (conversationData, etc). The object needs to be serializable to JSON and I don't believe a RegExp it is.
The workaround would be to store the regex as an string and then recreate the regex object using the constructor and the stored string expression.
Check the core-State sample to know more about the store capabilities.
I have been searching for hours, but I cannot find anything about this.
Situation:
Backend, existing of NodeJS + Express + Mongoose (+ MongoDB ofcourse).
Frontend retrieves object from the Backend.
Frontend makes some changes (adds/updates/removes some attributes).
Now I use mongoose: PersonModel.findByIdAndUpdate(id, updatedPersonObject);
Result: added properties are added. Updated properties are updated. Removed properties... are still there!
Now I've been searching for an elegant way to solve this, but the best I could come up with is something like:
var properties = Object.keys(PersonModel.schema.paths);
for (var i = 0, len = properties.length; i < len; i++) {
// explicitly remove values that are not in the update
var property = properties[i];
if (typeof(updatedPersonObject[property]) === 'undefined') {
// Mongoose does not like it if I remove the _id property
if (property !== '_id') {
oldPersonDocument[property] = undefined;
}
}
}
oldPersonDocument.save(function() {
PersonModel.findByIdAndUpdate(id, updatedPersonObject);
});
(I did not even include trivial code to fetch the old document).
I have to write this for every Object I want to update. I find it hard to believe that this is the best way to handle this. Any suggestions anyone?
Edit:
Another workaround I found: to unset a value in MongoDB you have to set it to undefined.
If I set this value in the frontend, it is lost in the REST-call. So I set it to null in the frontend, and then in the backend I convert all null-values to undefined.
Still ugly though. There must be a better way.
You could use replaceOne() if you want to know how many documents matched your filter condition and how many were changed (I believe it only changes one document, so this may not be useful to know). Docs: https://mongoosejs.com/docs/api/model.html#model_Model.replaceOne
Or you could use findOneAndReplace if you want to see the document. I don't know if it is the old doc or the new doc that is passed to the callback; the docs say Finds a matching document, replaces it with the provided doc, and passes the returned doc to the callback., but you could test that on your own. Docs: https://mongoosejs.com/docs/api.html#model_Model.findOneAndReplace
So, instead of:
PersonModel.findByIdAndUpdate(id, updatedPersonObject);, you could do:
PersonModel.replaceOne({ _id: id }, updatedPersonObject);
As long as you have all the properties you want on the object you will use to replace the old doc, you should be good to go.
Also really struggling with this but I don't think your solution is too bad. Our setup is frontend -> update function backend -> sanitize users input -> save in db. For the sanitization part, we use a helper function where we integrate your approach.
private static patchModel(dbDocToUpdate: IModel, dataFromUser: Record<string, any>): IModel {
const sanitized = {};
const properties = Object.keys(PersonModel.schema.paths);
for (const key of properties) {
if (key in dbDocToUpdate) {
sanitized[key] = data[key];
}
}
Object.assign(dbDocToUpdate, sanitized);
return dbDocToUpdate;
}
That works smoothly and sets the values to undefined. Hence, they get removed from the document in the db.
The only problem that remains for us is that we wanted to allow partial updates. With that solution that's not possible and you always have to send everything to the backend.
EDIT
Another workaround we found is setting the property to an empty string in the frontend. Mongo then also removes the property in the database
This may be a vary bad idea, or a possible solution that we have to a database concurrency problem.
We have a method that is called to do an update of a mongo record. We are seeing some concurrency problems - process A reads the record, process B reads the record, process A makes mods and saves the record, process makes B mods and saves the record. Because B reads after A, before A writes, it doesn't know about the changes A made, and we lose the data from A.
I'm wondering if we could not use a database semaphore, basically a field on the collection, that is a boolean. If we read the record at the start of the method, and the field is true, it's being edited. At that point, re-call the method using process.nexttick(), with the same data. Otherwise, set the semaphore, and carry on.
There would still be a bit of time between the read and the save, but it should be/could be faster than what we are doing now.
Be something like this. Any thoughts, anyone done anything like this? Will it even work?
function remove_source(service_id,session, next)
{
var User = Mongoose.model("User");
/* get the user, based on the session user id */
User.findById(session.me,function(err,user_info)
{
if (user_info.semaphore === true)
{
process.nextTick(remove_source(service_id,session,next));
}
else
{
user_info.semaphore = true;
user_info.save(function(err,user_new)
{
if (err) next(err,user_new);
else continue_on(null,user_new);
});
}
function continue_on(user_new)
{
etc.......
}
Edit: New Code:
The function now looks as follows. I'm doing individual updates to the arrays. This of course means that I now have the possibility, if the transaction fails between the first and second transactions, of having data out of sync. I'm thinking that I could simply resave the user object that I retrieved on entry into the function, overwriting my changes. I don't know if Mongoose/Mongo will not do the save if I have not changed that object, will have to try and see. Any more thoughts?
var User = Mongoose.model("User");
/* get the user, based on the session user id */
User.findById(session.me,function(err,user_info)
{
if (err)
{
next(err,user_info,null);
return;
}
if (!user_info || user_info.length === 0)
{
next(_e("ACCOUNT_NOT_FOUND"),"user_id: " + session.me);
return;
}
var source_service_info = _.where(user_info.credentials, {"source_service_id": service_id});
var source_service = source_service_info.source_service;
User.findByIdAndUpdate(session.me,{$pull: {"credentials": {"source_service_id": service_id}}},{},function(err,user_credential_removed)
{
if (err)
{
next(err,user_info,null);
return;
}
User.findByIdAndUpdate(session.me,{$pull: {"criteria": {"source_service": source_service}}},{},function(err,user_criteria_removed)
{
if (err)
{
next(err,user_info,null);
return;
}
else
{
next(null,user_criteria_removed);
}
});
});
});
};
The problem with your approach is that it just shortens the time during which the data could be read by a second process, it doesn't eliminate the problem.
The solution to this would be to set your semaphore in the same action as the read. I haven't used Mongoose, but in MongoDB you can use findAndModify to only return a User record if the semaphore is false, and if it is false, in one atomic operation, set the semaphore to true.
If you don't want to use findAndModify, you could first do an update that sets the semaphore true (or to some specific ID value so you know that it is YOUR semaphore) only if the semaphore is not set. Then, if that process succeeds, you could do the find (perhaps passing your semaphore ID as a criterion in the find). However, findAndModify, if it is available in Mongoose, would do that in one step.
A variation of that is described here: http://docs.mongodb.org/manual/tutorial/isolate-sequence-of-operations/ where you do a form of optimistic locking that checks that the old values are unchanged before changing them to the new values.
There is a variation on this that uses a separate table to simulate a two-phase commit: http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/
Edited: Upon interchange below, this seems to be a schema and updating issue. Question may become something like: I have some entries in an array, and the ordinal index to those entries relates to some other arrays as well. How do I perform deletes without having mismatches?
Three off the top possibilities occur, depending on frequency in the real world vs QA test scenarios.
Consider adding a deleted flag but keeping the records in the same order. If someone toggles, reuse the same record, but fix however you want.
Use an associative array (JS object) for each element (not a feature from relational world.) If you need an order, add an array that lists the keys in order. Both have syntax to update without touching anything other that what has changed, and will not overwrite changes to different fields.
Use an associative array where the keys are numbers. Actual deletion won't hurt retrieval.
stuff = {}
stuff[1] = {some:'details'}
stuff[2] = {some:'details2'}
Was
1) Are you making changes to the same field? Make that into an array, and push changes, and pop the latest to read the current value.
2) Are you changing different fields, but data is getting trounced? Then there is better syntax to use for the updating. you can update field by field.
$set: { 'fielda': 'valuea' }
won't lose edits on previous fields
3) change your schema
4) change the timing on the processes so they don't overlap. Or so they do so in smaller subsets, that you can manage to prevent from overlapping.
I'd like to know, just out of interest, what multiple processes are needed to make updates on the same record? I don't work with anything that looks like that.
I am building my first sailsjs and nodejs application, and it great :)
My situation, I have about 100 tables with the same stucture, I would like to decide "on the fly" which table to load.
my first thought was use somehow a dynamic class names. But I dont know how to do this with nodejs, maybe some one have an idea.
So I would create 100 "modelName".js files in my models folder.
I can use this in browser
window["fileName"].find()....
But I don't have any window object in nodejs
Second idea was to pass the tableName to the model, the problem is, I have to reinit the model, don't know how.
Any solutions?
Found a solution
var modelName = req.param('p');
this[modelName].find()...
Own answer by author is correct, but I will add something just for people who will use it in the future - you can get modelName from req.options.model when you are using Blueprints.
Unfortunately you can't use this[modelName] as option is giving you model name starting with small letter, so first you have to upper case first letter with e.g. var modelName = req.options.model.charAt(0).toUpperCase() + req.options.model.slice(1);
and then you are free to use this[modelName].whateverYouNeed
I used it for generic policy to let user editing only his own group elements.
var modelName = req.options.model.charAt(0).toUpperCase() + req.options.model.slice(1)
var elementID = null
if (req.params.id) { // To handle DELETE, PUT
elementID = req.params.id
}
if (req.body.id) { // To handle POST
elementID = req.body.id
}
this[modelName].findOne({
id: elementID
}).exec(function(err, contextElement) {
if(err) {
return res.serverError(err)
}
if(contextElement.group=== req.user.group.id) {
sails.log('accessing own: ' + modelName)
return next()
}
else {
return res.forbidden('Tried to access not owned object')
}
})
An alternative:
sails.models[Model].findOne({...})
Make sure to have your "Model" name as string in lowercase. It works like accessing a property inside an object
Another option that worked for me:
var modelName = "User";
global[modelName].find()....