How do I copy entries from one collection to another using mongoose? - node.js

I'm trying to create a little task management site for a work project. The overall goal is here is that the tasks stay the same each month (their status can be updated and whatnot), and they need to be duplicated at the start of each new month so they can be displayed and sorted by on a table.
I already figured out how to schedule the task, I have the table I need set up. A little explanation before the code - the way I'm planning on doing this is having two different task collections - one I've called "assignments", will have the tasks that need to be duplicated (with their description, status and other necessary data) and another collection, which I called "tasks", will have the exact same data but with an additional "date" field. This is where the table will get it's data from, the date is just for sorting purposes.
This is what I have so far -
Index.js: gets all the assignments from the database, and sends the object over to the duplicate function.
router.get('/test', async function(req, res, next) {
let allTasks = await dbModule.getAllAssignments();
let result = await dbModule.duplicateTasks(allTasks);
res.json(result);
});
dbmodule.js:
getAllAssignments: () => {
allAssignments = Assignment.find({});
return allAssignments;
},
duplicateTasks: (allTasksToAdd) => {
try {
for (let i = 0; i < allTasksToAdd.length; i++) {
let newTask = new Task({
customername: allTasksToAdd.customername,
provname: allTasksToAdd.provname,
description: allTasksToAdd.description,
status: allTasksToAdd.status,
date: "07-2020"
})
newTask.save();
}
return "Done"
} catch (error) {
return "Error"
}
}
The issue arises when I try and actually duplicate the tasks. For testing purposes I've entered the date manually this time, but that's all that ends up being inserted - just the date, the rest of the data is skipped. I've heard of db.collection.copyTo(), but I'm not sure if it'll allow me to insert the field I need or if it's supported in mongoose. I know there's absolutely an easier way to do this but I can't quite figure it out. I'd love some input and suggestions if anyone has any.
Thanks.

The problem is that allTasksToAdd.customername (and the other fields your trying to access) will be undefined. You need to access the fields under the current index:
let newTask = new Task({
customername: allTasksToAdd[i].customername,
provname: allTasksToAdd[i].provname,
description: allTasksToAdd[i].description,
status: allTasksToAdd[i].status,
date: "07-2020"
})
Note that you can simplify this by using a for .. of loop instead:
for (const task of allTasksToAdd) {
const newTask = new Task({
customername: task.customername,
provname: task.provname,
description: task.description,
status: task.status,
date: "07-2020"
});
newTask.save();
}

Related

Increment the Sort Key dynamoDB node.js

I am new to dynamodb.
I want to increment the Sort Key
If the id=0 the next id=1 and so on,
If the user(Partition key), id(Sort Key) add items the next add items the id increment 1.
The code use on PutItem with dynamodb.
Is possible to do that?
I did not want use the UUID( unique Key)
Most situations don't need an auto-incrementing attribute and DynamoDB doesn't provide this feature out of the box. This is considered to be an anti-pattern in distributed systems.
But, see How to autoincrement in DynamoDB if you really need to.
I understand that you may need this number because it is a legal obligation to have incremental invoice numbers for example.
One way would be to create a table to store your number sequences.
Add fields like:
{
name: "invoices",
prefix: "INV",
numberOfDigits: 5,
leasedValue: 1,
appliedValue: 1,
lastUpdatedTime: '2022-08-05'
},
{
name: "deliveryNotes",
prefix: "DN",
numberOfDigits: 5,
leasedValue: 1,
appliedValue: 1,
lastUpdatedTime: '2022-08-05'
}
You need 2 values (a lease and an applied value), to make sure you never skip a beat, even when things go wrong.
That check-lease-apply-release/rollback logic looks as follows:
async function useSequence(name: string, cb: async (uniqueNumber: string) => void) {
// 1. GET THE SEQUENCE FROM DATABASE
const sequence = await getSequence("invoices");
this.validateSequence(sequence);
// 2. INCREASE THE LEASED VALUE
const oldValue = sequence.appliedValue;
const leasedValue = oldValue + 1;
sequence.leasedValue = leasedValue;
await saveSequence(sequence);
try {
// 3. CREATE AND SAVE YOUR DOCUMENT
await cb(format(leasedValue));
// 4. INCREASE THE APPLIED VALUE
sequence.appliedValue++;
await saveSequence(sequence);
} catch(err) {
// 4B. ROLLBACK WHEN THINGS ARE BROKEN
console.err(err)
try {
const sequence = await getSequence(name);
sequence.leasedValue--;
this.validateSequence(sequence);
await saveSequence(sequence);
} catch (err2) {
console.error(err2);
}
throw err;
}
}
function validateSequence(sequence) {
// A CLEAN STATE, MEANS THAT THE NUMBERS ARE IN SYNC
if (sequence.leasedValue !== sequence.appliedValue) {
throw new Error("sequence is broken.");
}
}
Then, whenever you need a unique number you can use the above function to work in a protected scope, where the number will be rollbacked when something goes wrong.
const details = ...;
await useSequence("invoice", async (uniqueNumber) => {
const invoiceData = {...details, id: uniqueNumber};
const invoice = await this.createInvoice(invoiceData);
await this.saveInvoice(invoice);
})
Can it scale? Can it run on multiple instances? No, it can't. It never will be, because in most countries it's just not legal to do so. You're not allowed to send out invoice 6 before invoice 5 or to cancel invoice 5 after you've send invoice 6.
The only exception being, if you have multiple sequences. e.g. in some cases you're allowed to have a sequence per customer, or a sequence per payment system, ... Hence, you want them in your database.

Make one column dependent on association Sequelize

I have a table called HOUSE. And it has a column named STATUS.
I also have a table called TASK and it also has a column named STATUS.
Each house has many tasks. And if there's one task that has a status of inProgress, the house status shall be inProgress. And if all of the tasks are done, then house is done.
I want this status column of the house be dependent on the status of its all tasks.
When I call /getHouses, here's what I do to add a property called status to each house object, because currently I have no STATUS column in the HOUSE table.
exports.getMyHouses = (req, res) => {
const page = myUtil.parser.tryParseInt(req.query.page, 0)
const limit = myUtil.parser.tryParseInt(req.query.limit, 10)
db.House.findAndCountAll({
where: { userId: req.user.id },
include: [
{
model: db.Task,
as: "task",
include: [
{
model: db.Photo,
as: "photos"
}
]
},
{
model: db.Address,
as: "address"
}
],
offset: limit * page,
limit: limit,
order: [["id", "ASC"]],
})
.then(data => {
let newData = JSON.parse(JSON.stringify(data))
const houses = newData.rows
for (let house of houses) {
house.status = "done"
const tasks = house.task
for (let task of tasks) {
if (task.status == "inProgress") {
house.status = "inProgress"
break
}
}
}
res.json(myUtil.response.paging(newData, page, limit))
})
.catch(err => {
console.log("Error get houses: " + err.message)
res.status(500).send({
message: "An error has occured while retrieving data."
})
})
}
EDIT: I just realized that perhaps I can update the house's status column each time there's an update in the task's status. I've never thought about this before.
But I would still love it if anyone could confirm that this is a good strategy or if there's a better one.
The option you have is viable as long as filtering by the house's status isn't something you require. This would essentially be called a virtual field (since it isn't something directly from the database). If you do need to filter by this field, you'd then need to query for all the tasks InProgress and get the unique house IDs.
You could update the house's status column on task update too but you could run into some race conditions if, for example, multiple requests were being made to update tasks to the same house. Make sure to run a transaction here if you were too. Querying/filtering for houses with InProgress tasks would be much faster since you can query it directly. However, updates would be slower since you'd need to run a task update, a count query on tasks, and an update query on the house.
Both have it's pro's and con's, it mainly depends on your application design's requirement.

MongoDB and Mongoose - Creating custom, autoindexed secondary IDs?

I initialize my DB in the usual way:
mongoose.connect(`mongodb://uname:pword#127.0.0.1:port/dbname?authSource=admin`, {useNewUrlParser: true, autoIndex: false});
And I have a Schema, something like:
var materialSchema = new Schema({
bookID: {type: String, required: true},
active: Boolean,
name: {type: String, required: true},
stockLength: {type: Number, required: true}
});
module.exports = mongoose.model('material', materialSchema);
When I create a new material and add it to the database, it is automatically assigned the usual _id - which is a behaviour I want to maintain. BUT, I'd also like for bookID to be a unique, auto-incrementing index. This is for physical shelf storage, and not for queries or anything like that.
I'd like for bookID to increment in the following way:
A-001
A-002
A-003
...
A-098
A-099
A-100
B-001
...
B-100
...
Z-001
...
Z-100
In case the pattern above isn't clear, the pattern starts at A-001 and ultimately ends at Z-100. Each letter goes from 001 through 100 before moving to the next letter. Each new collection entry is just the next ID in the pattern. It is unlikely that the end will ever be reached, but we'll cross that bridge when we get there.
I've only ever used the default _id for indexing, and can't figure out how to make this pattern.
Thanks for any insight!
Edit #1
The best solution I've come up with so far is to have a separate .txt file with all of the IDs listed in order. As each new object is created, pop (... shift) the next ID off the top of the file. This might also have the added benefit of easily adding additional IDs at a later date. This will probably be the approach I take, but I'm still interested in the mongoose solution requested above.
Edit #2
So I think the solution I'm going to use is a little different. Basically, findOne sorted by bookID descending. Then use the value returned to set the next.
Material.findOne()
.sort({bookID : -1})
.exec((err, mat) => {
if(err) {
// Send error
}else if(!mat) {
// First bookID
}else {
// Indexes exist...
let nextId = getNextID(mat.bookID);
// ...
}
});
Still easy to modify getNextID() to add new/different IDs in the future (if/when "Z100" is reached)
Thanks again!
Ok, so to expand a little bit on Edit #2, I've come up with the following solution.
Within the model (schema) file, we add a schema pre() middleware, that executes when .save() is called, before the save occurs:
// An arrow function will not work on this guy, if you want to use the "this" keyword
materialSchema.pre('save', function(next) {
this.model('material').findOne() // Don't forget the .model(...) bit!
.sort({bookID : -1}) // All I need is the highest (i.e. most recent) bookID
.select('bookID') // Ditto above (not really necessary)
.exec((err, result) => {
if(err) {
return next(err); // Oopsies, an error!
}else if(!result) {
this.bookID = 'A-001'; // The case when collection is empty
}else {
this.bookID = getNextID(result.bookID); // Otherwise, increment ID
}
next(); // Don't forget this sucker! This is how you save
});
});
And that's about it! It isn't an in-built solution direct from Mongoose, but it works a treat.
Just for completeness, the getNextID function looks like:
function getNextID(curID) {
let letter = curID.split('-')[0];
let number = parseInt(curID.split('-')[1]);
if(number >= 100) { // Increase the letter and reset the number
letter = String.fromCharCode(letter.charCodeAt(0) + 1)
number = '001';
}else { // Only increase the number
number = ('' + (number + 1)).padStart(3, '0'); // Makes sure the numbers are always 3 digits long
}
return `${letter}-${number}`;
}
This'll do just dandy for now. Until we get to Z100. But I'll cross that bridge if/when it comes. No big deal at all.
And you don't need to do anything special to use it. Just save a new doc as normal, and it automatically fires:
new Material({
// New material properties
}).save((err, mat) => {
// Handle errors and returns ...
});

How to avoid two concurrent API requests breaking the logic behind document validation?

I have an API that in order to insert a new item it needs to be validated. The validation basically is a type validator(string, number, Date, e.t.c) and queries the database that checks if the "user" has an "item" in the same date, which if it does the validation is unsuccessful.
Pseudocode goes like this:
const Item = require("./models/item");
function post(newDoc){
let errors = await checkForDocErrors(newDoc)
if (errors) {
throw errors;
}
let itemCreated = await Item.create(newDoc);
return itemCreated;
}
My problem is if I do two concurrent requests like this:
const request = require("superagent");
// Inserts a new Item
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Water Bottle"
})
/*
Inserts a new Item, which shouldn't do. Resulting in two items having the
same date.
*/
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Toothpick"
})
Both will be successful, which it shouldn't be since an "user" cannot have two "items" in the same date.
If I execute the second one after the first is finished, everything works as expected.
request.post('http://127.0.0.1:5000/api/item') // Inserts a new Item
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Water Bottle"
})
.then((res) => {
// It is not successful since there is already an item with that date
// as expected
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Toothpick"
})
})
To avoid this I send one request with an array of documents, but I want to prevent this issue or at least make less likely to happen.
SOLUTION
I created a redis server. Used the package redis-lock and wrapped around the POST route.
var client = require("redis").createClient()
var lock = require("redis-lock")(client);
var itemController = require('./controllers/item');
router.post('/', function(req, res){
let userId = "";
if (typeof req.body === 'object' && typeof req.body.id_user === 'string') {
userId = req.body.id_user;
}
lock('POST ' + req.path + userId, async function(done){
try {
let result = await itemController.post(req.body)
res.json(result);
} catch (e) {
res.status(500).send("Server Error");
}
done()
})
}
Thank you.
Explain
That is a race condition.
two or more threads can access shared data and they try to change it at the same time
What is a race condition?
Solution:
There are many ways to prevent conflict data in this case, a lock is 1 option.
You can lock on application level or database level... but I prefer you read this thread before chose any of them.
Optimistic vs. Pessimistic locking
Quick solution: pessimistic-lock https://www.npmjs.com/package/redis-lock
You should create a composite index or a composite primary key that includes the id_user and the start_date fields. This will ensure that no documents for the same user with the same date can be created, and the database will throw an error if you'll try to do it.
Composite index with mongoose
You could also use transactions. To do it, you should execute the find and the create methods inside a transaction, to ensure that no concurrent queries on the same document will be executed.
Mongoose transactions tutorial
More infos
I would go with an unique composite index, that in your specific case should be something like
mySchema.index({user_id: 1, start_date: 1}, {unique: true});

Nested transactions with pg-promise

I am using NodeJS, PostgreSQL and the amazing pg-promise library. In my case, I want to execute three main queries:
Insert one tweet in the table 'tweets'.
In case there is hashtags in the tweet, insert them into another table 'hashtags'
Them link both tweet and hashtag in a third table 'hashtagmap' (many to many relational table)
Here is a sample of the request's body (JSON):
{
"id":"12344444",
"created_at":"1999-01-08 04:05:06 -8:00",
"userid":"#postman",
"tweet":"This is the first test from postman!",
"coordinates":"",
"favorite_count":"0",
"retweet_count":"2",
"hashtags":{
"0":{
"name":"test",
"relevancetraffic":"f",
"relevancedisaster":"f"
},
"1":{
"name":"postman",
"relevancetraffic":"f",
"relevancedisaster":"f"
},
"2":{
"name":"bestApp",
"relevancetraffic":"f",
"relevancedisaster":"f"
}
}
All the fields above should be included in the table "tweets" besides hashtags, that in turn should be included in the table "hashtags".
Here is the code I am using based on Nested transactions from pg-promise docs inside a NodeJS module. I guess I need nested transactions because I need to know both tweet_id and hashtag_id in order to link them in the hashtagmap table.
// Columns
var tweetCols = ['id','created_at','userid','tweet','coordinates','favorite_count','retweet_count'];
var hashtagCols = ['name','relevancetraffic','relevancedisaster'];
//pgp Column Sets
var cs_tweets = new pgp.helpers.ColumnSet(tweetCols, {table: 'tweets'});
var cs_hashtags = new pgp.helpers.ColumnSet(hashtagCols, {table:'hashtags'});
return{
// Transactions
add: body =>
rep.tx(t => {
return t.one(pgp.helpers.insert(body,cs_tweets)+" ON CONFLICT(id) DO UPDATE SET coordinates = "+body.coordinates+" RETURNING id")
.then(tweet => {
var queries = [];
for(var i = 0; i < body.hashtags.length; i++){
queries.push(
t.tx(t1 => {
return t1.one(pgp.helpers.insert(body.hashtags[i],cs_hashtags) + "ON CONFLICT(name) DO UPDATE SET fool ='f' RETURNING id")
.then(hash =>{
t1.tx(t2 =>{
return t2.none("INSERT INTO hashtagmap(tweetid,hashtagid) VALUES("+tweet.id+","+hash.id+") ON CONFLICT DO NOTHING");
});
});
}));
}
return t.batch(queries);
});
})
}
The problem is with this code I am being able to successfully insert the tweet but nothing happens then. I cannot insert the hashtags nor link the hashtag to the tweets.
Sorry but I am new to coding so I guess I didn't understood how to properly return from the transaction and how to perform this simple task. Hope you can help me.
Thank you in advance.
Jean
Improving on Jean Phelippe's own answer:
// Columns
var tweetCols = ['id', 'created_at', 'userid', 'tweet', 'coordinates', 'favorite_count', 'retweet_count'];
var hashtagCols = ['name', 'relevancetraffic', 'relevancedisaster'];
//pgp Column Sets
var cs_tweets = new pgp.helpers.ColumnSet(tweetCols, {table: 'tweets'});
var cs_hashtags = new pgp.helpers.ColumnSet(hashtagCols, {table: 'hashtags'});
return {
/* Tweets */
// Add a new tweet and update the corresponding hash tags
add: body =>
db.tx(t => {
return t.one(pgp.helpers.insert(body, cs_tweets) + ' ON CONFLICT(id) DO UPDATE SET coordinates = ' + body.coordinates + ' RETURNING id')
.then(tweet => {
var queries = Object.keys(body.hashtags).map((_, idx) => {
return t.one(pgp.helpers.insert(body.hashtags[i], cs_hashtags) + 'ON CONFLICT(name) DO UPDATE SET fool = $1 RETURNING id', 'f')
.then(hash => {
return t.none('INSERT INTO hashtagmap(tweetid, hashtagid) VALUES($1, $2) ON CONFLICT DO NOTHING', [+tweet.id, +hash.id]);
});
});
return t.batch(queries);
});
})
.then(data => {
// transaction was committed;
// data = [null, null,...] as per t.none('INSERT INTO hashtagmap...
})
.catch(error => {
// transaction rolled back
})
},
NOTES:
As per my notes earlier, you must chain all queries, or else you will end up with loose promises
Stay away from nested transactions, unless you understand exactly how they work in PostgreSQL (read this, and specifically the Limitations section).
Avoid manual query formatting, it is not safe, always rely on the library's query formatting.
Unless you are passing the result of transaction somewhere else, you should at least provide the .catch handler.
P.S. For the syntax like +tweet.id, it is the same as parseInt(tweet.id), just shorter, in case those are strings ;)
For those who will face similar problem, I will post the answer.
Firstly, my mistakes:
In the for loop : body.hashtag.length doesn't exist because I am dealing with an object (very basic mistake here). Changed to Object.keys(body.hashtags).length
Why using so many transactions? Following the answer by vitaly-t in: Interdependent Transactions with pg-promise I removed the extra transactions. It's not yet clear for me how you can open one transaction and use the result of one query into another in the same transaction.
Here is the final code:
// Columns
var tweetCols = ['id','created_at','userid','tweet','coordinates','favorite_count','retweet_count'];
var hashtagCols = ['name','relevancetraffic','relevancedisaster'];
//pgp Column Sets
var cs_tweets = new pgp.helpers.ColumnSet(tweetCols, {table: 'tweets'});
var cs_hashtags = new pgp.helpers.ColumnSet(hashtagCols, {table:'hashtags'});
return {
/* Tweets */
// Add a new tweet and update the corresponding hashtags
add: body =>
rep.tx(t => {
return t.one(pgp.helpers.insert(body,cs_tweets)+" ON CONFLICT(id) DO UPDATE SET coordinates = "+body.coordinates+" RETURNING id")
.then(tweet => {
var queries = [];
for(var i = 0; i < Object.keys(body.hashtags).length; i++){
queries.push(
t.one(pgp.helpers.insert(body.hashtags[i],cs_hashtags) + "ON CONFLICT(name) DO UPDATE SET fool ='f' RETURNING id")
.then(hash =>{
t.none("INSERT INTO hashtagmap(tweetid,hashtagid) VALUES("+tweet.id+","+hash.id+") ON CONFLICT DO NOTHING");
})
);
}
return t.batch(queries);
});
}),

Resources