how to bulk update using knexjs - node.js

I'm trying to batch update table users that contains these columns id (primary key) , status (text) , active (numeric).
the array i'm receiving from back-end is something like this:
[
{ id: 33715, status: 'online', active: 10 },
{ id: 39129, status: 'offline', active: 0.1 },
{ id: 36090, status: 'loggedin', active: 24 },
{ id: 34452, status: 'loggedout', active: 1 },
]
active is time in hours. now i want to bulk update this array into users table. as each object represents a row in a table.
I've tried this approach according to this solution Patrick Motard
function bulkUpdate (records) {
var updateQuery = [
'INSERT INTO users (id, status, active) VALUES',
_.map(records, () => '(?)').join(','),
'ON DUPLICATE KEY UPDATE',
'status = VALUES(status),',
'active = VALUES(active)'
].join(' '),
vals = [];
_(records).map(record => {
vals.push(_(record).values());
});
return knex.raw(updateQuery, vals)
.catch(err => {
console.log(err)
});
}
bulkUpdate(response);
but i get this error
error: syntax error at or near "DUPLICATE"
so what i'm missing here. and does anyone by chance have a better solution without using promises or bluebird then do trx.commit , this consumes large cpu and ram. and doesn't do the purpose of update 10,000 row at once

I don't see any ON DUPLICATE KEY UPDATE reference in
https://www.postgresql.org/docs/9.5/sql-insert.html
You could try out ON CONFLICT DO UPDATE SET. Try out first with plain SQL to write a query that seems to work correctly and then it is easy to convert to javascript generated one.

Related

How do I copy entries from one collection to another using mongoose?

I'm trying to create a little task management site for a work project. The overall goal is here is that the tasks stay the same each month (their status can be updated and whatnot), and they need to be duplicated at the start of each new month so they can be displayed and sorted by on a table.
I already figured out how to schedule the task, I have the table I need set up. A little explanation before the code - the way I'm planning on doing this is having two different task collections - one I've called "assignments", will have the tasks that need to be duplicated (with their description, status and other necessary data) and another collection, which I called "tasks", will have the exact same data but with an additional "date" field. This is where the table will get it's data from, the date is just for sorting purposes.
This is what I have so far -
Index.js: gets all the assignments from the database, and sends the object over to the duplicate function.
router.get('/test', async function(req, res, next) {
let allTasks = await dbModule.getAllAssignments();
let result = await dbModule.duplicateTasks(allTasks);
res.json(result);
});
dbmodule.js:
getAllAssignments: () => {
allAssignments = Assignment.find({});
return allAssignments;
},
duplicateTasks: (allTasksToAdd) => {
try {
for (let i = 0; i < allTasksToAdd.length; i++) {
let newTask = new Task({
customername: allTasksToAdd.customername,
provname: allTasksToAdd.provname,
description: allTasksToAdd.description,
status: allTasksToAdd.status,
date: "07-2020"
})
newTask.save();
}
return "Done"
} catch (error) {
return "Error"
}
}
The issue arises when I try and actually duplicate the tasks. For testing purposes I've entered the date manually this time, but that's all that ends up being inserted - just the date, the rest of the data is skipped. I've heard of db.collection.copyTo(), but I'm not sure if it'll allow me to insert the field I need or if it's supported in mongoose. I know there's absolutely an easier way to do this but I can't quite figure it out. I'd love some input and suggestions if anyone has any.
Thanks.
The problem is that allTasksToAdd.customername (and the other fields your trying to access) will be undefined. You need to access the fields under the current index:
let newTask = new Task({
customername: allTasksToAdd[i].customername,
provname: allTasksToAdd[i].provname,
description: allTasksToAdd[i].description,
status: allTasksToAdd[i].status,
date: "07-2020"
})
Note that you can simplify this by using a for .. of loop instead:
for (const task of allTasksToAdd) {
const newTask = new Task({
customername: task.customername,
provname: task.provname,
description: task.description,
status: task.status,
date: "07-2020"
});
newTask.save();
}

How do I count the documents that include a value within an array?

I have a Mongoose abTest document that has two fields:
status. This is a string enum and can be of type active, inactive or draft.
validCountryCodes. This is an array of strings enums (GB, EU, AU etc). By default, it will be empty.
In the DB, at any one time, I only want there to be one active abTest for each validCountryCode so I'm performing some validation prior to creating or editing a new abTest.
To do this, I've written a function that attempts to count the number of documents that have a status of active and that contain one of the countryCodes.
The function will then return if the count is more than one. If so, I will throw a validation error.
if (params.status === 'active') {
const activeTestForCountryExists = await checkIfActiveAbTestForCountry(
validCountryCodes,
);
if (params.activeTestForCountryExists) {
throw new ValidationError({
message: 'There can only be one active test for each country code.',
});
}
}
const abTest = await AbTest.create(params);
checkIfActiveAbTestForCountry() looks like this:
const checkIfActiveAbTestForCountry = async countryCodes => {
const query = {
status: 'active',
};
if (
!countryCodes ||
(Array.isArray(countryCodes) && countryCodes.length === 0)
) {
query.validCountryCodes = {
$eq: [],
};
} else {
query.validCountryCodes = { $in: [countryCodes] };
}
const count = await AbTest.countDocuments(query);
return count > 0;
};
The count query should count not only exact array matches, but for any partial matches.
If in the DB there is an active abTest with a validCountryCodes array of ['GB', 'AU',], the attempting to create a new abTest with ['GB' should fail. As there is already a test with GB as a validCountryCode.
Similarly, if there is a test with a validCountryCodes array of ['AU'], then creating a test with validCountryCodes of ['AU,'NZ'] should also fail.
Neither is enforced right now.
How can I do this? Is this possible write a query that checks for this?
I considered iterating over params.validCountryCodes and counting the docs that include each, but this seems like bad practice.
take a look at this MongoDB documantation.
As I understood what you need is to find out if there is any document that contains at least one of the specified countryCodes and it has active status. then your query should look like this:
{
status: 'active',
$or: [
{ validCountryCodes: countryCodes[0] },
{ validCountryCodes: countryCodes[1] },
// ...
]
}
note that counting documents is not an efficient manner to check if a document exists or not, instead use findOne with only one field being projected.
You are using the correct mongo-query for your requirement. Can you verify the actual queries executed from your application is the same? Check here
{ status: 'active', validCountryCodes: { $in: [ countryCodes ] } }
For eg; below query :
{ status: 'active', validCountryCodes: { $in: ['GB' ] } }
should match document :
{ status: 'active', validCountryCodes: ['AU','GB'] }

Make one column dependent on association Sequelize

I have a table called HOUSE. And it has a column named STATUS.
I also have a table called TASK and it also has a column named STATUS.
Each house has many tasks. And if there's one task that has a status of inProgress, the house status shall be inProgress. And if all of the tasks are done, then house is done.
I want this status column of the house be dependent on the status of its all tasks.
When I call /getHouses, here's what I do to add a property called status to each house object, because currently I have no STATUS column in the HOUSE table.
exports.getMyHouses = (req, res) => {
const page = myUtil.parser.tryParseInt(req.query.page, 0)
const limit = myUtil.parser.tryParseInt(req.query.limit, 10)
db.House.findAndCountAll({
where: { userId: req.user.id },
include: [
{
model: db.Task,
as: "task",
include: [
{
model: db.Photo,
as: "photos"
}
]
},
{
model: db.Address,
as: "address"
}
],
offset: limit * page,
limit: limit,
order: [["id", "ASC"]],
})
.then(data => {
let newData = JSON.parse(JSON.stringify(data))
const houses = newData.rows
for (let house of houses) {
house.status = "done"
const tasks = house.task
for (let task of tasks) {
if (task.status == "inProgress") {
house.status = "inProgress"
break
}
}
}
res.json(myUtil.response.paging(newData, page, limit))
})
.catch(err => {
console.log("Error get houses: " + err.message)
res.status(500).send({
message: "An error has occured while retrieving data."
})
})
}
EDIT: I just realized that perhaps I can update the house's status column each time there's an update in the task's status. I've never thought about this before.
But I would still love it if anyone could confirm that this is a good strategy or if there's a better one.
The option you have is viable as long as filtering by the house's status isn't something you require. This would essentially be called a virtual field (since it isn't something directly from the database). If you do need to filter by this field, you'd then need to query for all the tasks InProgress and get the unique house IDs.
You could update the house's status column on task update too but you could run into some race conditions if, for example, multiple requests were being made to update tasks to the same house. Make sure to run a transaction here if you were too. Querying/filtering for houses with InProgress tasks would be much faster since you can query it directly. However, updates would be slower since you'd need to run a task update, a count query on tasks, and an update query on the house.
Both have it's pro's and con's, it mainly depends on your application design's requirement.

Multiple findOneAndUpdate operations are being skipped

I have a forEach loop where I am querying a document, doing some simple math calculations and then updating a document in the collection and move on to the next iteration.
The problem is, alot of times randomly some of the UPDATE operations will not update the document. I don't know why is it happening. Is it because of the lock?
I have tried logging things just before the update operation. The data is all correct but when it comes to update, it will randomly not update at all. Out of 10 iterations, lets say 8 will correctly work
const name = "foo_game";
players.forEach(({ id, team, username }) => {
let updatedStats = {};
Users.findOne({ id }).then(existingPlayer => {
if (!existingPlayer) return;
const { stats } = existingPlayer;
const existingStats = stats[pug.name];
if (!existingStats) return;
const presentWins = existingStats.won || 0;
const presentLosses = existingStats.lost || 0;
updatedStats = {
...existingStats,
won:
team === winningTeam
? presentWins + 1
: changeWinner
? presentWins - 1
: presentWins,
lost:
team !== winningTeam
? presentLosses + 1
: changeWinner
? presentLosses - 1
: presentLosses,
};
// THE CALCULATIONS ARE ALL CORRECT TILL THIS POINT
// THE UPDATE WIILL RANDOMLY NOT WORK
Users.findOneAndUpdate(
{ id, server_id: serverId },
{
$set: {
username,
stats: { ...stats, [name]: updatedStats },
},
},
{
upsert: true,
}
).exec();
});
});
Basically what you are missing here is the asynchronous operations of both the findOne() and the findOneAndUpdate() are not guaranteed to complete before your foreach() is completed. Using forEach() is not a great choice for a loop with async operations in it, but the other main point here is that it's completely unnecessary since MongoDB has a much better way of doing this and in one request to the server.
In short, instead of "looping" you actually want to provide an array of instructions to bulkWrite():
let server_id = serverId; // Alias one of your variables or just change it's name
Users.bulkWrite(
players.map(({ id, team, username }) =>
({
"updateOne": {
"filter": { _id, server_id },
"update": {
"$set": { username },
"$inc": {
[`stats.${name}.won`]:
team === winningTeam ? 1 : changeWinner ? - 1 : 0,
[`stats.${name}.lost`]:
team !== winningTeam ? 1 : changeWinner ? - 1 : 0
}
},
"upsert": true
}
})
)
)
.then(() => /* whatever continuation here */ )
.catch(e => console.error(e))
So rather than looping, that Array.map() produces one "updateOne" statement within the bulk operation for each array member and sends it to the server. The other change of course is you simply do not need the findOne() in order to read existing values. All you really need is to use the $inc operator in order to either increase or decrease the current value. Noting here that if nothing is currently recorded at the specified path, then it would create those with whatever value of 1/-1/0 was determined by the logic and handed to $inc.
Note this is how you actually should be doing things in general, as aside from avoiding uneccesary loops of async calls the main thing here is to actually use the atomic operators like $inc that MongoDB has. Reading data state from the database in order to make changes is an anti-pattern and best avoided.

Mongodb duplicate errorr

db.collection("resource").update({name: name}, {
name: name,
type: type
}, {
upsert: true
}
I differentiate documents by their names. I do not add document if it exists with the same. But I want to warn user by saying "It already exists, operation failed" How can I achieve it?
It sounds like you want to insert documents, not update or insert documents.
1: Add unique index on resource.name ahead of time.
db.resouces.createIndex({ name: 1 }, { unique: true })
Important: do this once, not on every request.
See mongodb create index docs.
2: Use insert instead of update + upsert.
It sounds like you want to actually insert a document, and get an error if there is a duplicate key.
db.resources.insert({ name: "AJ" }) // ok
db.resources.insert({ name: "AJ" }) // error!
You will get a duplicate key error on the second insert. Error code 11000.
See mongodb docs on insert.
3: Use promise-try-catch in javascript.
The code to do error checking looks like:
var db = require("mongojs")(DATABASE_URL, [ "resources" ])
var duplicateKey = function (err) {
return err.code == "11000"
}
db.resources.insert({ name: name })
.then(function () {
// success!
})
.catch(duplicateKey, function () {
// sorry! name is taken
})

Resources