I have
db.result('DELETE FROM categories WHERE id = ${id}', category).then(function (data) { ...
and
db.many('SELECT * FROM categories').then(function (data) { ...
initially delete is called from one API call and then select on following API call, but callback for db request happens in reverse order, so I get list of categories with removed category.
Is there a way how to lock categories table with pg-promise?
If you want the result of the SELECT to always reflect the result of the previous DELETE, then you have two approaches to consider...
The standard approach is to unify the operations into one, so you end up executing all your dependent queries against the same connection:
db.task(function * (t) {
yield t.none('DELETE FROM categories WHERE id = ${id}', category);
return yield t.any('SELECT FROM categories');
})
.then(data => {
// data = only the categories that weren't deleted
});
You can, of course, also use either the standard promise syntax or even ES7 await/async.
The second approach would be to organize an artificial lock inside your service that would hold off on executing any corresponding SELECT until the DELETE requests are all done.
However, this is a very awkward solution, typically pointing at the flaw in the architecture. Also, as the author of pg-promise, I won't be even getting into that solution, as it would be way outside of my library anyway.
Related
I've found numerous examples of nest 'example' apps, but each one seems to have slightly different opinions on design patterns.
I'm currently interested in where the object preparation work should go between a resolver and service when coupled with TypeORM.
for example:
comment.resolver.ts:
/********************
* #MUTATION
*********************/
/**
*
* #param payload
* #param args
*/
#Mutation('createComment')
async create(#CurrentUser() payload: JwtPayload, #Args('input') args: CreateCommentInputDto): Promise < CommentEntity > {
const currentUser = await this.userService.getCurrentUser(payload);
const initComment = new CommentEntity();
const newComment: CommentEntity = {
...args,
createdBy: currentUser,
createdDate: new Date(),
modifiedDate: new Date(),
...initComment,
};
const createdComment = await this.commentService.create(newComment);
pubSub.publish('CommentCreated', {
CommentCreated: createdComment
});
return createdComment;
}
comment.service.ts:
/**
*
* #param comment
*/
async create(comment: CommentEntity): Promise < CommentEntity > {
return await this.CommentsRepository.save(comment);
}
i.e.
Create new empty comment Entity
Add field values that are not supplied by query
use spread operator to combine them all together
pass them all to the comment service to save via TypeORM repository
The reasoning being that the comment service just accepts and saves a well formatted entity. Maybe in the future I i will need to prepare the comment to be created in a different way, and so that would be defined in a new mutation.
Is this an anti-pattern? Should I be moving that object create / combine / formatting into the service, and keep the resolver method as light as possible?
If so, what's the logic behind that?
You should check the preload method that is provided by the Repository item provided by TypeOrm. It allows batching changes on an existing entity or new one, which should be what you want.
I think that TypeOrm is very unopiniated, you are free to choose how you organise your mutations on entities. Still I think the 'preload' repository pattern is a safe one as you always want first to get value from the database corresponding to your proposed changes, then you batch the changes in the entity and save it afterwards. It should lower your chances of getting a conflict value on an entity or getting double values etc.
You can see the database as a git repository, fetch first, rebase your local changes on the remote head value, commit and push your changes.
This is a really trivial problem. I am just curious on how to deal with this in a "professional" manner.
I am trying to stick to variable naming convention. For NodeJs I am doing camelCasing. For database, I am using PostgreSQL and using underscore_casing.
Now the problem arises when I query data from PostgreSQL. I'll get a user object with following format,
{user_id: 1, account_type : "Admin"}
I can pass this object directly to server side-render and will have to use underscore casing to access account_type. Of course, I can manually create a new user JSON object with property userId and accountType but that is unnecessary work.
Is it possible to follow variable naming convention for both language and avoid having mixed variable names casing in some files? What is a good way to stay organized?
The are two good ways to approach this issue. The simplest one - do no conversion, use the exact database names. And the second one is to camel-case columns automatically.
Either way, you should always follow the underscore notation for all PostgreSQL declarations, as it will give you the option to activate camel-casing in your app at a later time, if it becomes necessary. Never use camel-case inside the database, or you will end up in a lot of pain later.
If you want the best of both worlds, follow the underscore notation for all PostgreSQL declarations, and convert to camel-case as you read data.
Below is an example of how to do it properly with pg-promise, copied from event receive example:
// Example below shows the fastest way to camelize column names:
const options = {
receive(e) {
camelizeColumns(e.data);
}
};
function camelizeColumns(data) {
const template = data[0];
for (var prop in template) {
const camel = pgp.utils.camelize(prop);
if (!(camel in template)) {
for (var i = 0; i < data.length; i++) {
const d = data[i];
d[camel] = d[prop];
delete d[prop];
}
}
}
}
Also see the following article: Pg-promise and case sensitivity in column names.
UPDATE
The code above has been updated for use of pg-promise v11 or later.
I've struggled with this too, and I've concluded that there's really no way to avoid this kind of ugliness unless you rewrite the objects that come from the database. Fortunately, that's not too difficult in Javascript:
const fromDBtoJS = (obj) => {
// declare a variable to hold the result
const result = {};
// iterate over the keys on the object
Object.keys(obj).forEach((key) => {
// adjust the key
const newKey = key.replace(/_[a-z]/g, (x) => x[1].toUpperCase());
// add the value from the old object with the new key
result[newKey] = obj[key];
});
// return the result
return result;
};
Here's a JSFiddle. The "replace" code above was found here
If you wanted to use classes for models in your application, you could incorporate this code into the constructor or database load method so it's all handled more-or-less automatically.
I'm using MongoDB with NodeJS. Therefore I use mongoose.
I'm developing a multi player real time game. So I receive many requests from many players sometimes at the very same time.
I can simplify it by saying that I have a house collection, that looks like this:
{
"_id" : 1,
"items": [item1, item2, item3]
}
I have a static function, called after each request is received:
house.statics.addItem = function(id, item, callback){
var HouseModel = this;
HouseModel.findById(id, function(err, house){
if (err) throw err;
//make some calculations such as:
if (house.items.length < 4){
HouseModel.findByIdAndUpdate(id, {$push: {items: item}}, cb);
}
});
}
In this example, I coded so that the house document can never have more than 4 items. But what happens is that when I receive several request at the very same time, this function is executed twice by both requests and since it is asynchronous, they both push a new item to the items field and then my house has 5 items.
I am doing something wrong? How can I avoid that behavior in the future?
yes, you need better locking on the houseModel, to indicate that an addItem
is in progress.
The problem is that multiple requests can call findById and see the same
house.items.length, then each determine based on that (outdated) snapshot
that it is ok to add one more item. The nodejs boundary of atomicity is the
callback; between an async call and its callback, other requests can run.
One easy fix is to track not just the number of items in the house but the
number of intended addItems as well. On entry into addItem, bump the "want
to add more" count, and test that.
One possible approach since the release of Mongoose 4.10.8 is writing a plugin which makes save() fail if the document has been modified since you loaded it. A partial example is referenced in #4004:
#vkarpov15 said:
8b4870c should give you the general direction of how one would write a plugin for this
Since Mongoose 4.10.8, plugins now have access to this.$where. For documents which have been loaded from the database (i.e., are not this.isNew), the plugin can add conditions which will be evaluated by MongoDB during the update which can prevent the update from actually happening. Also, if a schema’s saveErrorIfNotFound option is enabled, the save() will return an error instead of succeeding if the document failed to save.
By writing such a plugin and changing some property (such as a version number) on every update to the document, you can implement “optimistic concurrency” (as #4004 is titled). I.e., you can write code that roughly does findOne(), do some modification logic, save(), if (ex) retry(). If all you care about is a document remaining self-consistent and ensuring that Mongoose’s validators run and your document is not highly contentious, this lets you write code that is simple (no need to use something which bypasses Mongoose’s validators like .update()) without sacrificing safety (i.e., you can reject save()s if the document was modified in the meantime and avoid overwriting committed changes).
Sorry, I do not have a code example yet nor do I know if there is a package on npm which implements this pattern as a plugin yet.
I am also building a multiplayer game and ran into the same issue. I believe I have solved it my implementing a queue-like structure:
class NpcSaveQueue {
constructor() {
this.queue = new Map();
this.runQueue();
}
addToQueue(unitId, obj) {
if (!this.queue.has(unitId)) {
this.queue.set(String(unitId), obj);
} else {
this.queue.set(String(unitId), {
...this.queue.get(unitId),
...obj,
})
}
}
emptyUnitQueue(unitId) {
this.queue.delete(unitId);
}
async executeUnitQueue(unitId) {
await NPC.findByIdAndUpdate(unitId, this.queue.get(unitId));
this.emptyUnitQueue(unitId);
}
runQueue() {
setInterval(() => {
this.queue.forEach((value, key) => {
this.executeUnitQueue(key);
})
}, 1000)
}
}
Then when I want to update an NPC, instead of interacting with Mongoose directly, I run:
npcSaveQueue.addToQueue(unit._id, {
"location.x": newLocation.x,
"location.y": newLocation.y,
});
That way, every second, the SaveQueue just executes all code for every NPC that requires updating.
This function never executes twice, because update operation is atomic on a level of single document.
More info in official manual: http://docs.mongodb.org/manual/core/write-operations-atomicity/#atomicity-and-transactions
I'm new to key value stores, but I'd like to learn. As a personal project, I'm trying to build an inventory management system with Node.js and Redis. Let's assume this is the correct technology to use for the moment.
If I had a simple system, that needs to track the number of widgets at a particular location, with the ability to look up details by widget or by location, my understanding according to https://matt.sh/thinking-in-redis-part-one is to store separate "custom indexes" to look up by location and by item.
In node.js to save a new entry then, we would create the entry with hmset, add the entry to the 2 indexes with sadd:
redis.hmset([
key,
'attr1', entry.attr1,
'attr2', entry.attr2,
'attr3', entry.attr3,
],
function(err) {
// add entry to location set
var locationKey = 'location:' + entry.location;
redis.sadd(locationKey, key, function(err) {
redis.expire(locationKey, 900);
});
// add entry to widget set
var widgetKey = 'widget:' + widget.id;
redis.sadd(widgetKey, key, function(err) {
redis.expire(widgetKey, 900);
});
redis.expire(key, 900);
}
);
Now if we wanted to move all widgets from one location to another, we'd need to get all entries from the widget set, add the entry to the new location index, and remove it from the old index:
// move all widgets to another location
redis.smembers('widget:' + widget.id, function(err, entryKeys) {
entryKeys.forEach(function(entryKey) {
// get entry for rebroadcast
redis.hgetall(entryKey, function(err, result) {
if (result) {
// add entry to new location's index
var locationKey = 'location:' + location;
redis.sadd(locationKey, entryKey, function(err) {
redis.expire(locationKey, 900);
});
// remove entry from old location's index
redis.srem('location:' + result.location, entryKey);
}
});
});
});
My concern is the number of requests that need to be made for each command. Adding an entry, will cost 3 inserts for the data itself, and 3 more assuming we want to expire the data. Moving all widgets will require 1+n inserts, n reads, and n deletes.
If this were for a real time game with hundreds or thousands of requests a second, is it ok for each command to require this many calls? Is this normal for a redis implementation?
Yes.
Redis is that fast. But do a benchmark on your machine, or similar production machine, that will run redis. It is included in redis itself.. (Post it back here, I'd be interested as well.)
Redis has a lot of commands at its disposal, and your data organization might allow cheaper calls, or calls-less-often. That will depend how you lay out the data model. There is not really a "query language" like SQL that can do lots of stuff in the query or combine queries into a single one. You are meant to hit redis a lot, which is a different philosophy than SQL (to some extent).
This personal project will allow you to see what works and what could be made better, so kudos on the effort. Good luck!
This may be a vary bad idea, or a possible solution that we have to a database concurrency problem.
We have a method that is called to do an update of a mongo record. We are seeing some concurrency problems - process A reads the record, process B reads the record, process A makes mods and saves the record, process makes B mods and saves the record. Because B reads after A, before A writes, it doesn't know about the changes A made, and we lose the data from A.
I'm wondering if we could not use a database semaphore, basically a field on the collection, that is a boolean. If we read the record at the start of the method, and the field is true, it's being edited. At that point, re-call the method using process.nexttick(), with the same data. Otherwise, set the semaphore, and carry on.
There would still be a bit of time between the read and the save, but it should be/could be faster than what we are doing now.
Be something like this. Any thoughts, anyone done anything like this? Will it even work?
function remove_source(service_id,session, next)
{
var User = Mongoose.model("User");
/* get the user, based on the session user id */
User.findById(session.me,function(err,user_info)
{
if (user_info.semaphore === true)
{
process.nextTick(remove_source(service_id,session,next));
}
else
{
user_info.semaphore = true;
user_info.save(function(err,user_new)
{
if (err) next(err,user_new);
else continue_on(null,user_new);
});
}
function continue_on(user_new)
{
etc.......
}
Edit: New Code:
The function now looks as follows. I'm doing individual updates to the arrays. This of course means that I now have the possibility, if the transaction fails between the first and second transactions, of having data out of sync. I'm thinking that I could simply resave the user object that I retrieved on entry into the function, overwriting my changes. I don't know if Mongoose/Mongo will not do the save if I have not changed that object, will have to try and see. Any more thoughts?
var User = Mongoose.model("User");
/* get the user, based on the session user id */
User.findById(session.me,function(err,user_info)
{
if (err)
{
next(err,user_info,null);
return;
}
if (!user_info || user_info.length === 0)
{
next(_e("ACCOUNT_NOT_FOUND"),"user_id: " + session.me);
return;
}
var source_service_info = _.where(user_info.credentials, {"source_service_id": service_id});
var source_service = source_service_info.source_service;
User.findByIdAndUpdate(session.me,{$pull: {"credentials": {"source_service_id": service_id}}},{},function(err,user_credential_removed)
{
if (err)
{
next(err,user_info,null);
return;
}
User.findByIdAndUpdate(session.me,{$pull: {"criteria": {"source_service": source_service}}},{},function(err,user_criteria_removed)
{
if (err)
{
next(err,user_info,null);
return;
}
else
{
next(null,user_criteria_removed);
}
});
});
});
};
The problem with your approach is that it just shortens the time during which the data could be read by a second process, it doesn't eliminate the problem.
The solution to this would be to set your semaphore in the same action as the read. I haven't used Mongoose, but in MongoDB you can use findAndModify to only return a User record if the semaphore is false, and if it is false, in one atomic operation, set the semaphore to true.
If you don't want to use findAndModify, you could first do an update that sets the semaphore true (or to some specific ID value so you know that it is YOUR semaphore) only if the semaphore is not set. Then, if that process succeeds, you could do the find (perhaps passing your semaphore ID as a criterion in the find). However, findAndModify, if it is available in Mongoose, would do that in one step.
A variation of that is described here: http://docs.mongodb.org/manual/tutorial/isolate-sequence-of-operations/ where you do a form of optimistic locking that checks that the old values are unchanged before changing them to the new values.
There is a variation on this that uses a separate table to simulate a two-phase commit: http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/
Edited: Upon interchange below, this seems to be a schema and updating issue. Question may become something like: I have some entries in an array, and the ordinal index to those entries relates to some other arrays as well. How do I perform deletes without having mismatches?
Three off the top possibilities occur, depending on frequency in the real world vs QA test scenarios.
Consider adding a deleted flag but keeping the records in the same order. If someone toggles, reuse the same record, but fix however you want.
Use an associative array (JS object) for each element (not a feature from relational world.) If you need an order, add an array that lists the keys in order. Both have syntax to update without touching anything other that what has changed, and will not overwrite changes to different fields.
Use an associative array where the keys are numbers. Actual deletion won't hurt retrieval.
stuff = {}
stuff[1] = {some:'details'}
stuff[2] = {some:'details2'}
Was
1) Are you making changes to the same field? Make that into an array, and push changes, and pop the latest to read the current value.
2) Are you changing different fields, but data is getting trounced? Then there is better syntax to use for the updating. you can update field by field.
$set: { 'fielda': 'valuea' }
won't lose edits on previous fields
3) change your schema
4) change the timing on the processes so they don't overlap. Or so they do so in smaller subsets, that you can manage to prevent from overlapping.
I'd like to know, just out of interest, what multiple processes are needed to make updates on the same record? I don't work with anything that looks like that.