I'm trying to batch together a database update to re-populate a mongo collection. I've created an object to hold the properties needed to lookup the data from an external source, and then add it back to a MongoDb collection.
The array looks like this:
const pops = [
{ table: 'SFAccounts',
label: 'Account__c',
createListName: 'Accounts'
},
{ table: 'SFTimes',
label: 'CusTime__c',
createListName: 'Time'
}]
I want to then create a function that takes 'table', 'label, and 'createListName' and it does something basically like this..
async function processData(table, label, createListName) {
// Get some info from Salesforce
const dataFromSF = await getMetaDataFromSalesForce(table)
// Extract the parts I actually need
const relevantBits = dataFromSF.filter(field => field.name === label)
//Create a new list in the db
const createResult = await List.create( { name: createListName, values: relevantBits } )
return createResult
}
The end goal is to get to something like
await Promise.all(processData(pops))
Which will await all the tables being pulled and populated into the database.
If you change the args of processData:
async function processData({table, label, createListName}) {
// Get some info from Salesforce
const dataFromSF = await getMetaDataFromSalesForce(table)
// Extract the parts I actually need
const relevantBits = dataFromSF.filter(field => field.name === label)
//Create a new list in the db
const createResult = await List.create( { name: createListName, values: relevantBits } )
return createResult
}
it's just await Promise.all(pops.map(processData));
Related
This is my first time of using bulkWrite to carry out updates via mongoose. I am building a blog application and I am using it to learn MERN stack. I have a Post model. The Post model has object value which is an array. This is an example of it:
const PostSchema = new mongoose.Schema(
{
postLikes:{
type: Array,
default: []
}
}
)
The postLikes contain mongodb object ids of users who liked a post.
I have a logic for deleting selected users or all users by an admin. The like system does not come with a Like Model of it own. I simply used an array system inside the post model. After deleting a user, I would like to update all post models with likes of the selected users. Some users may have multiple likes across different posts.
In my node, I created a variable like this:
const {selectedIds} = req.body;
The selectedIds came from reactjs like this:
const [selectedUsers, setSelectedUsers] = useState([]);
const arrayOfSelectedUserId = (userId) =>{
setSelectedUsers(prevArray => [...prevArray, userId]);
);
}
For the request, I did it like this:
const response = await axiosPrivate.post(`/v1/users/deleteSelected`, selectedIds, { withCredentials: true,
headers:{authorization: `Bearer ${auth.token}`}})
In nodejs, the selectedUsers ids was passed to this variable:
const {selectedIds} = req.body;
I created the logic this way:
const findIntersection = (array1, array2) => {
return array1.filter((elem) => {
return array2.indexOf(elem) !== -1;
});
}
const filteredPost = posts.filter((singleFilter) => {
const intersection = findIntersection(selectedIds, singleFilter.postLikes);
return singleFilter.postLikes.length !== 0 && intersection.length !== 0;
});
const updatedPosts = filteredPost.map((obj)=>{
const intersection = findIntersection(selectedIds, obj.postLikes);
console.log(intersection )
return {
updateOne: {
filter: { _id: obj._id },
update: { $pull: { postLikes: { $in: intersection } } },
},
};
});
Post.bulkWrite(updatedPosts).then((res) => {
console.log("Documents Updated", res.modifiedCount)
})
The console.log shows the text Document updated and showed number of documents updated. However, if I check my database, the update won't reflect. This means that the selected users' ID is still in the array.
Is there a better method? What Am I doing wrong?
I have to read a really large CSV file so search through the google and get to know about createReadStream. I am using a program that read the csv file data and insert it into the mongoDB.
process I am following
process the data using createReadStream (I think it read the file line by line).
Storing data into an array.
Insert the data into mongoDB using insertMany
Now the problem is whole file is first get stored into an array and then I insert into the database.
But what I think is the better approach would be I only store first 500 line/rows into an array insert it into the DB and again follow the same step for the next 500 records
Is it possible to achieve this ?
and also is it the right way to do this ?
my program
const test = async () => {
const stream = fs.createReadStream(workerData)
.pipe(parse())
.on('data', async function(csvrow) {
try{
stream.pause()
if(!authorName.includes(csvrow.author)) {
const author = new Author({author: csvrow.author})
authorId = author._id
authorName.push(author.author)
authorData.push(author)
}
if(!companyName.includes(csvrow.company_name)) {
const company = new Company({companyName: csvrow.company_name})
companyID = company._id
companyName.push(company.companyName)
companyData.push(company)
}
users = new User({
name: csvrow.firstname,
dob: csvrow.dob,
address: csvrow.address,
phone: csvrow.phone,
state: csvrow.state,
zip: csvrow.zip,
email: csvrow.email,
gender: csvrow.gender,
userType: csvrow.userType
})
userData.push(users)
book = new Book({
book_number: csvrow.book_number,
book_name: csvrow.book_name,
book_desc: csvrow.book_desc,
user_id: users._id,
author_id: authorId
})
bookData.push(book)
relationalData.push({
username: users.name,
author_id: authorId,
book_id: book._id,
company_id: companyID
})
}finally {
stream.resume()
}
})
.on('end', async function() {
try {
Author.insertMany(authorData)
User.insertMany(userData)
Book.insertMany(bookData)
Company.insertMany(companyData)
await Relational.insertMany(relationalData)
parentPort.postMessage("true")
}catch(e){
console.log(e)
parentPort.postMessage("false")
}
})
}
test()
This program is working fine also inserting the data into the DB, But I am looking for something like this:
const stream = fs.createReadStream(workerData)
.pipe(parse())
.on('data', async function(csvrow, maxLineToRead: 500) {
// whole code/logic of insert data into DB
})
so maxLineToRead is my imaginary term.
basically my point is I want to process 500 data at a time and insert it into the DB and want to repeat this process till the end.
You can create a higher scoped array variable where you accumulate rows of data as they arrive on the data event. When you get to 500 rows, fire off your database operation to insert them. If not yet at 500 rows, then just add the next one to the array and wait for more data events to come.
Then, in the end event insert any remaining rows still in the higher scoped array.
In this way, you will insert 500 at a time and then however many are left at the end. This has an advantage vs. inserting them all at the end that you spread out the database load over the time you are parsing.
Here's an attempt to implement that type of processing. There are some unknowns (documented with comments) based on an incomplete description of exactly what you're trying to accomplish in some circumstances):
const test = () => {
return new Promise((resolve, reject) => {
const accumulatedRows = [];
async function processRows(rows) {
// initialize data arrays that we will insert
const authorData = [],
companyData = [],
userData = [],
bookData = [],
relationalData = [];
// this code still has a problem that I don't have enough context
// to know how to solve
// If authorName contains csvrow.author, then the variable
// authorId is not initialized, but is used later in the code
// This is a problem that needs to be fixed.
// The same issue occurs for companyID
for (let csvrow of rows) {
let authorId, companyID;
if (!authorName.includes(csvrow.author)) {
const author = new Author({ author: csvrow.author })
authorId = author._id
authorName.push(author.author)
authorData.push(author)
}
if (!companyName.includes(csvrow.company_name)) {
const company = new Company({ companyName: csvrow.company_name })
companyID = company._id
companyName.push(company.companyName)
companyData.push(company)
}
let users = new User({
name: csvrow.firstname,
dob: csvrow.dob,
address: csvrow.address,
phone: csvrow.phone,
state: csvrow.state,
zip: csvrow.zip,
email: csvrow.email,
gender: csvrow.gender,
userType: csvrow.userType
});
userData.push(users)
let book = new Book({
book_number: csvrow.book_number,
book_name: csvrow.book_name,
book_desc: csvrow.book_desc,
user_id: users._id,
author_id: authorId
});
bookData.push(book)
relationalData.push({
username: users.name,
author_id: authorId,
book_id: book._id,
company_id: companyID
});
}
// all local arrays of data are populated now for this batch
// so add this data to the database
await Author.insertMany(authorData);
await User.insertMany(userData);
await Book.insertMany(bookData);
await Company.insertMany(companyData);
await Relational.insertMany(relationalData);
}
const batchSize = 50;
const stream = fs.createReadStream(workerData)
.pipe(parse())
.on('data', async function(csvrow) {
try {
accumulatedRows.push(csvRow);
if (accumulatedRows.length >= batchSize) {
stream.pause();
await processRows(accumulatedRows);
// clear out the rows we just processed
acculatedRows.length = 0;
stream.resume();
}
} catch (e) {
// calling destroy(e) will prevent leaking a stream
// and will trigger the error event to be called with that error
stream.destroy(e);
}
}).on('end', async function() {
try {
await processRows(accumulatedRows);
resolve();
} catch (e) {
reject(e);
}
}).on('error', (e) => {
reject(e);
});
});
}
test().then(() => {
parentPort.postMessage("true");
}).catch(err => {
console.log(err);
parentPort.postMessage("false");
});
I have a node API which connects to a mongoDB through mongoose. I am creating an advanced results middleware that enabled selecting, filtering, sorting, pagination etc. based on a Brad Traversy course Node.js API Masterclass With Express & MongoDB. This is all good.
I am adapting the code from the course to be able to use the $ne (not equal) operator and I want to be able to get a model that is not equal to a nested property (user id) of the model. I am using this for an explore feature to see a list of things, but I don't want to show the user their own things. I am having trouble figuring out how to access the id property.
********************* UPDATE *********************
It seems all the documentation I've read recommends writing const injected like this:
const injected = {
'user._id': { "$ne": req.user.id }
};
but for some reason it is not working. I can query top level properties that are just a plain string value like this:
const injected = {
access: { "$ne": "public" }
};
but not a property on an object. Does anyone know why? Is it because the property I want to query is an id? I've also tried:
const injected = {
'user._id': { "$ne": mongoose.Types.ObjectId(req.user.id) }
};
which also does not work...
So the model looks like this:
{
name: 'Awesome post',
access: 'public',
user: {
_id: '2425635463456241345', // property I want to access
}
}
then the actual advanced results middleware looks like this and it's the 'injected' object where I am trying to access id. In the course brad uses this syntax to use lte (/?averageCost[lte]=10000) but I do not get any results with my ne. Can anyone help me here?
const advancedResults = (model, populate) => async (req, res, next) => {
let query;
const injected = {
access: 'public',
'user._id[ne]': req.user.id, // I don't think user._id[ne] is correct
};
}
// Copy req.query
const reqQuery = { ...req.query, ...injected };
console.log('injected: ', injected);
// Fields to exclude
const removeFields = ['select', 'sort', 'page', 'limit'];
// Loop over removeFields and delete them from reqQuery
removeFields.forEach(param => delete reqQuery[param]);
// Create query string
let queryStr = JSON.stringify(reqQuery);
// Create operators ($gt, $gte, etc)
queryStr = queryStr.replace(/\b(gt|gte|lt|lte|in|ne)\b/g, match => `$${match}`);
// Finding resource and remove version
query = model.find(JSON.parse(queryStr)).select('-__v');
// Select Fields
if (req.query.select) {
const fields = req.query.select.split(',').join(' ');
query = query.select(fields);
}
// Sort
if (req.query.sort) {
const sortBy = req.query.sort.split(',').join(' ');
query = query.sort(sortBy);
} else {
query = query.sort('-createdAt');
}
// Pagination
const page = parseInt(req.query.page, 10) || 1;
const limit = parseInt(req.query.limit, 10) || 25;
const startIndex = (page - 1) * limit;
const endIndex = page * limit;
const total = await model.countDocuments(JSON.parse(queryStr));
query = query.skip(startIndex).limit(limit);
if (populate) {
query = query.populate(populate);
}
// Executing query
const results = await query;
// Pagination result
const pagination = {};
if (endIndex < total) {
pagination.next = {
page: page + 1,
limit,
};
}
if (startIndex > 0) {
pagination.prev = {
page: page - 1,
limit,
};
}
res.advancedResults = {
success: true,
count: results.length,
pagination,
data: results,
};
next();
};
module.exports = advancedResults;
Answering your question about how to use $ne:
The use of $ne is as follows:
"field":{
"$ne": yourValue
}
Into your query should be like:
"user._id": {
"$ne": req.user.id
}
Example here
$ne operator will return all document where the field value don't match with the given value.
As you have done, to acces the nested field is necessary use the dot notation.
Also, to ensure it works, if your schema defines _id as ObjectId maybe is necessary parse req.user.id to ObjectId.
But if in your schema is a string then should works.
So try (not tested at all):
const injected = {
'user._id': { "$ne": req.user.id }
};
When I want to add two records in sequence, only one record is added, on the second it throws an error due to the fact that it cannot create a field with such data:
"NOTES_ID is required","key: NOTES_ID, value: undefined, is not a
number"
How to create an entry for two related tables sequentially from the beginning for the main table, and then for the one that has the foreign key installed.
module.exports.create = async function (req, res) {
const stateMatrix = await StateMatrix.select().exec()
const noteObj = {
DATE: req.body.DATE,
TITLE: req.body.TITLE,
CONTENT: req.body.CONTENT
};
const noteStateObj = {
STATE_DATE: new Date().toLocaleDateString("en-US"),
STATES_ID: stateMatrix[0]._props.STATES_ID_CURR,
NOTES_ID: req.body.NOTE_ID,
USERS_ID: req.decoded.user_id
};
try {
await Notes.create(noteObj);
await NoteStates.create(noteStateObj);
res.status(201).json(noteObj, noteStateObj);
} catch (e) {
errorHandler(res, e);
}
};
Probably NoteStates is related to Notes through note_id field which can not be empty (I guess it's foreign key). It means that you should set it before saving noteStateObj:
// Something like this
const newNote = await Notes.create(noteObj);
noteStateObj.NOTES_ID = newNote.ID;
await NoteStates.create(noteStateObj);
Let us assume that we have two collections say "users" and "usersList"
Upon creating a new user document in users collection with following object
{username: Suren, age:31}
The function should read the above data and update other collection i.e. "usersList" with the username alone like below
{username: Suren}
Let me know the possibility
The code I have tried is
exports.userCreated =
functions.firestore.document('users/{userId}').onCreate((event) => {
const post = event.data.data();
return event.data.ref.set(post, {merge: true});
})
I have done it using below code
exports.userCreated = functions.firestore.document('users/{userId}')
.onCreate((event) => {
const firestore = admin.firestore()
return firestore.collection('usersList').doc('yourDocID').update({
name:'username',
}).then(() => {
// Document updated successfully.
console.log("Doc updated successfully");
});
})
If all you want to do is strip the age property from the document, you can do it like this:
exports.userCreated = functions.firestore.document('users/{userId}').onCreate((event) => {
const post = event.data.data();
delete post.age;
return event.data.ref.set(post);
})